lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process. lsst.ctrl.bps.drivers INFO: Starting submission process lsst.ctrl.bps.drivers INFO: Initializing execution environment lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 4.5737 seconds; current memory usage: 0.178 Gibyte, delta: 0.041 Gibyte, peak delta: 0.054 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.190 Gibyte (main), 0.000 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph) lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.0067 seconds lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.0140 seconds lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/pipeline_orig.qgraph DEEP/20220825/B1b/science#step1/20240307T102204Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/pipeline.qgraph lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 5.1328 seconds lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 12.9721 seconds; current memory usage: 0.362 Gibyte, delta: 0.184 Gibyte, peak delta: 0.172 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.362 Gibyte (main), 0.348 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs) lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0172 seconds; current memory usage: 0.362 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.362 Gibyte (main), 0.348 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 558 cluster(s) lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow) lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20220825_B1b_science#step1_20240307T102204Z' lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 0.1195 seconds; current memory usage: 0.362 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.362 Gibyte (main), 0.348 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 560 job(s) (including final) lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow) parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2 Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses s_addresses.add(address_by_interface(interface)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper retval = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface return socket.inet_ntoa(fcntl.ioctl( ^^^^^^^^^^^^ OSError: [Errno 99] Cannot assign requested address lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 0.2484 seconds; current memory usage: 0.364 Gibyte, delta: 0.001 Gibyte, peak delta: 0.002 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.364 Gibyte (main), 0.348 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting submit stage lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000 parsl.dataflow.dflow INFO: Starting DataFlowKernel with config Config( app_cache=True, checkpoint_files=None, checkpoint_mode='task_exit', checkpoint_period=None, executors=[MultiHighThroughputExecutor()], garbage_collect=True, initialize_logging=True, internal_tasks_max_threads=10, max_idletime=120.0, monitoring=None, retries=1, retry_handler=None, run_dir='runinfo', strategy='simple', usage_tracking=False ) parsl.dataflow.dflow INFO: Parsl version: 2023.06.12 parsl.usage_tracking.usage DEBUG: Tracking status: False parsl.dataflow.dflow INFO: Run id is: e3f32546-8d2a-4e3b-86ac-bb002179551e parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name parsl.dataflow.memoization INFO: App caching initialized parsl.dataflow.strategy DEBUG: Scaling strategy: simple parsl.executors.high_throughput.executor DEBUG: Starting queue management thread parsl.executors.high_throughput.executor DEBUG: queue management worker starting parsl.executors.high_throughput.executor DEBUG: Started queue management thread Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z parsl.executors.high_throughput.executor DEBUG: Created management thread: parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id={block_id} --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider: parsl.executors.status_handling INFO: Scaling out by 1 blocks parsl.executors.status_handling INFO: Allocated block ID 0 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0 2024-03-07 10:22:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 2024-03-07 10:22:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:22:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-07 10:22:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:22:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local parsl.providers.local.local DEBUG: Launching in remote mode 2024-03-07 10:22:46 proc_lsst.multi:170 [INFO] [multi] job_id 24403 proc_lsst.multi INFO: [multi] job_id 24403 2024-03-07 10:22:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:22:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:22:46 proc_lsst.multi:178 [INFO] [multi] provider local accepted submit and returned 24403 proc_lsst.multi INFO: [multi] provider local accepted submit and returned 24403 parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 24403 parsl.dataflow.job_status_poller DEBUG: Adding executor multi parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 0 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 0 has memoization hash cd24ff0df55efc830f746ca70a2aa27a parsl.dataflow.memoization INFO: Task 0 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149813490c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1 parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/4bf76bac-1005-4152-a7d7-fcae9edc155e_isr_1125543_62.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/4bf76bac-1005-4152-a7d7-fcae9edc155e_isr_1125543_62.stderr parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1 submitted for App characterizeImage, waiting on task 0 parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2 submitted for App calibrate, waiting on task 1 parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 2 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3 has memoization hash e5d4e3f3f8aceb2a776bfefa1a563f40 parsl.dataflow.memoization INFO: Task 3 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134914e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 2 parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/805b5321-2360-462b-9ce0-27c562c7de95_isr_1125543_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/805b5321-2360-462b-9ce0-27c562c7de95_isr_1125543_19.stderr parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4 submitted for App characterizeImage, waiting on task 3 parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 4 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5 submitted for App calibrate, waiting on task 4 parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 5 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 6 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 6 has memoization hash f297b13547e88aae194fc8199094ade2 parsl.dataflow.memoization INFO: Task 6 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149813491760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 3 parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/851df404-9543-44bc-bc67-0ad5a76a7a17_isr_1125542_38.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/851df404-9543-44bc-bc67-0ad5a76a7a17_isr_1125542_38.stderr parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 7 submitted for App characterizeImage, waiting on task 6 parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 7 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 8 submitted for App calibrate, waiting on task 7 parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 8 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 9 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 9 has memoization hash 68c4b37e9c9a63d5269263144a945b41 parsl.dataflow.memoization INFO: Task 9 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149813491940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 4 parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/2318460f-9087-4c88-b4a3-161ab5f60bd1_isr_1125541_29.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/2318460f-9087-4c88-b4a3-161ab5f60bd1_isr_1125541_29.stderr parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 10 submitted for App characterizeImage, waiting on task 9 parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 10 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 11 submitted for App calibrate, waiting on task 10 parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 11 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 12 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 12 has memoization hash 79d417ecb2d231f2a4e5f56c25708069 parsl.dataflow.memoization INFO: Task 12 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149813491e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 5 parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/9ad16f3e-3405-4846-9779-aa429f5db5d0_isr_1125542_58.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/9ad16f3e-3405-4846-9779-aa429f5db5d0_isr_1125542_58.stderr parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 13 submitted for App characterizeImage, waiting on task 12 parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 13 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 14 submitted for App calibrate, waiting on task 13 parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 14 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 15 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 15 has memoization hash e6defd131ef8b524a75ff4e595fd606d parsl.dataflow.memoization INFO: Task 15 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149813492480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 6 parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/a960964f-61d7-46a4-94ca-acf35d2bb2e2_isr_1125542_60.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/a960964f-61d7-46a4-94ca-acf35d2bb2e2_isr_1125542_60.stderr parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 16 submitted for App characterizeImage, waiting on task 15 parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 16 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 17 submitted for App calibrate, waiting on task 16 parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 17 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 18 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 18 has memoization hash 9643916a9bfeae06d85f204837e9c343 parsl.dataflow.memoization INFO: Task 18 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134925c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 7 parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/df0137b0-06e9-48a0-9ce9-3a82db8f6302_isr_1125543_12.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/df0137b0-06e9-48a0-9ce9-3a82db8f6302_isr_1125543_12.stderr parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 19 submitted for App characterizeImage, waiting on task 18 parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 19 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 20 submitted for App calibrate, waiting on task 19 parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 20 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 21 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 21 has memoization hash 34c07959d23fbdadc5fd6290f3309cbc parsl.dataflow.memoization INFO: Task 21 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149813492840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 8 parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/e5faee53-0d84-4f16-993b-92d02b526756_isr_1125543_2.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/e5faee53-0d84-4f16-993b-92d02b526756_isr_1125543_2.stderr parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 22 submitted for App characterizeImage, waiting on task 21 parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 22 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 23 submitted for App calibrate, waiting on task 22 parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 23 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 24 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 24 has memoization hash c4a2b06e71e2e095ed74de4d588f4bc4 parsl.dataflow.memoization INFO: Task 24 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149813493ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 9 parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/c95b5212-b6b1-4eaa-bee2-a6160b9532c0_isr_1125541_26.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/c95b5212-b6b1-4eaa-bee2-a6160b9532c0_isr_1125541_26.stderr parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 25 submitted for App characterizeImage, waiting on task 24 parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 25 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 26 submitted for App calibrate, waiting on task 25 parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 26 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 27 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 27 has memoization hash 420f47f290fbdaf2e8a39a5f6f3f6f60 parsl.dataflow.memoization INFO: Task 27 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149813493880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 10 parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/fe61e302-b4c8-4664-ae50-5ccd6e0b34ee_isr_1125541_2.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/fe61e302-b4c8-4664-ae50-5ccd6e0b34ee_isr_1125541_2.stderr parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 28 submitted for App characterizeImage, waiting on task 27 parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 28 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 29 submitted for App calibrate, waiting on task 28 parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 29 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 30 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 30 has memoization hash b16c30ae4c93f4bbaea1865cde59f3af parsl.dataflow.memoization INFO: Task 30 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149813493560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 11 parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/aeb7b31f-c03d-4f4b-a5b6-100a8517add4_isr_1125542_50.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/aeb7b31f-c03d-4f4b-a5b6-100a8517add4_isr_1125542_50.stderr parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 31 submitted for App characterizeImage, waiting on task 30 parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 31 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 32 submitted for App calibrate, waiting on task 31 parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 32 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 33 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 33 has memoization hash 7c3e7c2e6b87ed31b3eeee700eb254dc parsl.dataflow.memoization INFO: Task 33 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149813493240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 12 parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/720ea0d2-368f-4438-adbc-62614c90467f_isr_1125543_15.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/720ea0d2-368f-4438-adbc-62614c90467f_isr_1125543_15.stderr parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 34 submitted for App characterizeImage, waiting on task 33 parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 34 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 35 submitted for App calibrate, waiting on task 34 parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 35 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 36 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 36 has memoization hash 5d452f89f8d73f2f3ee0fa236f624483 parsl.dataflow.memoization INFO: Task 36 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149813492f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 13 parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/18eee0f8-3c10-4585-99fb-8290c0838c2e_isr_1125543_50.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/18eee0f8-3c10-4585-99fb-8290c0838c2e_isr_1125543_50.stderr parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 37 submitted for App characterizeImage, waiting on task 36 parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 37 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 38 submitted for App calibrate, waiting on task 37 parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 38 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 39 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 39 has memoization hash 7c70c45f6ae22219c4ffa23239de091b parsl.dataflow.memoization INFO: Task 39 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149813492b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 14 parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/04fadd06-5556-45db-b61e-c451c61ce1a5_isr_1125541_59.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/04fadd06-5556-45db-b61e-c451c61ce1a5_isr_1125541_59.stderr parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 40 submitted for App characterizeImage, waiting on task 39 parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 40 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 41 submitted for App calibrate, waiting on task 40 parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 41 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 42 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 42 has memoization hash fec0bc1b672bd61b871e01de1a726613 parsl.dataflow.memoization INFO: Task 42 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149813493f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 15 parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/66a2283c-1d20-4b76-96d1-f5c38149f4ba_isr_1125543_8.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/66a2283c-1d20-4b76-96d1-f5c38149f4ba_isr_1125543_8.stderr parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 43 submitted for App characterizeImage, waiting on task 42 parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 43 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 44 submitted for App calibrate, waiting on task 43 parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 44 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 45 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 45 has memoization hash 2a7f2a7ff0a66ce83a5964eed8e4c6cf parsl.dataflow.memoization INFO: Task 45 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b42c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 16 parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/71312d8a-ed98-4a88-9154-7dae2eecac39_isr_1125542_40.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/71312d8a-ed98-4a88-9154-7dae2eecac39_isr_1125542_40.stderr parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 46 submitted for App characterizeImage, waiting on task 45 parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 46 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 47 submitted for App calibrate, waiting on task 46 parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 47 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 48 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 48 has memoization hash c2c5e53b074e93574caf73a3cdd687eb parsl.dataflow.memoization INFO: Task 48 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b45e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 17 parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/793bf2fe-d265-42cc-8e76-cffff8507a2a_isr_1125543_26.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/793bf2fe-d265-42cc-8e76-cffff8507a2a_isr_1125543_26.stderr parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 49 submitted for App characterizeImage, waiting on task 48 parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 49 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 50 submitted for App calibrate, waiting on task 49 parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 50 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 51 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 51 has memoization hash 3e6436c36d6e23be5bf3606e345cf427 parsl.dataflow.memoization INFO: Task 51 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b4900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 18 parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/c842de3a-cd6b-494e-9246-c31cb66bb608_isr_1125541_4.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/c842de3a-cd6b-494e-9246-c31cb66bb608_isr_1125541_4.stderr parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 52 submitted for App characterizeImage, waiting on task 51 parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 52 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 53 submitted for App calibrate, waiting on task 52 parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 53 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 54 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 54 has memoization hash d2fd7db9ae464f6735afc88eb8b4d776 parsl.dataflow.memoization INFO: Task 54 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b4c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 19 parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/c8ab690d-ed69-4612-83fb-1c8ad955eff9_isr_1125541_44.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/c8ab690d-ed69-4612-83fb-1c8ad955eff9_isr_1125541_44.stderr parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 55 submitted for App characterizeImage, waiting on task 54 parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 55 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 56 submitted for App calibrate, waiting on task 55 parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 56 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 57 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 57 has memoization hash eb7c6d93f9d19c091334eb8144448c9a parsl.dataflow.memoization INFO: Task 57 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b4f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 20 parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/39fe399f-2cc6-4269-92df-94dacd011566_isr_1125541_10.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/39fe399f-2cc6-4269-92df-94dacd011566_isr_1125541_10.stderr parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 58 submitted for App characterizeImage, waiting on task 57 parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 58 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 59 submitted for App calibrate, waiting on task 58 parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 59 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 60 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 60 has memoization hash 90d0cb34c2c5be1ae0056d03e945c4c5 parsl.dataflow.memoization INFO: Task 60 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b5260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 21 parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/2a6828e6-cd3a-4d51-93c0-bae7627e7e3b_isr_1125541_27.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/2a6828e6-cd3a-4d51-93c0-bae7627e7e3b_isr_1125541_27.stderr parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 61 submitted for App characterizeImage, waiting on task 60 parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 61 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 62 submitted for App calibrate, waiting on task 61 parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 62 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 63 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 63 has memoization hash 2cea4ee5f3ac67119cd796783c04cef9 parsl.dataflow.memoization INFO: Task 63 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b5580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 22 parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/9f0d5ca1-6d0e-49a8-a2b5-d6aebc13ab97_isr_1125543_38.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/9f0d5ca1-6d0e-49a8-a2b5-d6aebc13ab97_isr_1125543_38.stderr parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 64 submitted for App characterizeImage, waiting on task 63 parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 64 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 65 submitted for App calibrate, waiting on task 64 parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 65 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 66 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 66 has memoization hash 1cd77630fc2a9dbea05944e7a172ae55 parsl.dataflow.memoization INFO: Task 66 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b58a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 23 parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/fefae949-217e-4698-a2b3-85adf081ec0d_isr_1125541_32.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/fefae949-217e-4698-a2b3-85adf081ec0d_isr_1125541_32.stderr parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 67 submitted for App characterizeImage, waiting on task 66 parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 67 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 68 submitted for App calibrate, waiting on task 67 parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 68 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 69 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 69 has memoization hash 32bb69bedc6b42c627e501f2f8929f58 parsl.dataflow.memoization INFO: Task 69 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b5bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 24 parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/fe0f060f-4c5f-4a2b-a366-f9c734197777_isr_1125543_42.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/fe0f060f-4c5f-4a2b-a366-f9c734197777_isr_1125543_42.stderr parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 70 submitted for App characterizeImage, waiting on task 69 parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 70 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 71 submitted for App calibrate, waiting on task 70 parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 71 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 72 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 72 has memoization hash 5c7716922ebb550ddcf53f65f4d9f55b parsl.dataflow.memoization INFO: Task 72 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b5ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 25 parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/c111edf7-bf1b-49c3-8055-eb45d3d4cacb_isr_1125541_12.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/c111edf7-bf1b-49c3-8055-eb45d3d4cacb_isr_1125541_12.stderr parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 73 submitted for App characterizeImage, waiting on task 72 parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 73 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 74 submitted for App calibrate, waiting on task 73 parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 74 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 75 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 75 has memoization hash cff6d20bbc7d2e96fbe4393a27ab17c3 parsl.dataflow.memoization INFO: Task 75 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b6200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 26 parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/a3da51ab-b09f-4fd8-8f49-cd8f5d926a3e_isr_1125543_35.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/a3da51ab-b09f-4fd8-8f49-cd8f5d926a3e_isr_1125543_35.stderr parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 76 submitted for App characterizeImage, waiting on task 75 parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 76 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 77 submitted for App calibrate, waiting on task 76 parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 77 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 78 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 78 has memoization hash ebc1894eb45e5295bfec1f445c15e63b parsl.dataflow.memoization INFO: Task 78 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b6520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 27 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/6a09bc2f-ee6b-48a1-ab05-e07448768c06_isr_1125541_20.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/6a09bc2f-ee6b-48a1-ab05-e07448768c06_isr_1125541_20.stderr parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 79 submitted for App characterizeImage, waiting on task 78 parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 79 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 80 submitted for App calibrate, waiting on task 79 parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 80 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 81 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 81 has memoization hash fdab7ba8e09743eecf0a3e5a5dc57313 parsl.dataflow.memoization INFO: Task 81 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b6840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 28 parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/94c38760-f56c-4087-88fd-08ee197ffcd4_isr_1125542_34.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/94c38760-f56c-4087-88fd-08ee197ffcd4_isr_1125542_34.stderr parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 82 submitted for App characterizeImage, waiting on task 81 parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 82 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 83 submitted for App calibrate, waiting on task 82 parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 83 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 84 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 84 has memoization hash 6d4f1e636f7c7987e8f75f7f11bfbc6a parsl.dataflow.memoization INFO: Task 84 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b6b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 29 parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/00f0be9b-32ce-470f-a0eb-2fe550fba8c1_isr_1125543_32.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/00f0be9b-32ce-470f-a0eb-2fe550fba8c1_isr_1125543_32.stderr parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 85 submitted for App characterizeImage, waiting on task 84 parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 85 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 86 submitted for App calibrate, waiting on task 85 parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 86 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 87 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 87 has memoization hash 5268857d1e52a169354e897e39a29228 parsl.dataflow.memoization INFO: Task 87 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b6e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 30 parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/86198a6c-8b7a-45ac-971e-3712e8fda3c8_isr_1125541_33.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/86198a6c-8b7a-45ac-971e-3712e8fda3c8_isr_1125541_33.stderr parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 88 submitted for App characterizeImage, waiting on task 87 parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 88 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 89 submitted for App calibrate, waiting on task 88 parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 89 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 90 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 90 has memoization hash d8949447b080eaaff56134719c6a5134 parsl.dataflow.memoization INFO: Task 90 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b71a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 31 parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/194df398-5cff-484f-be9c-cc1fe89b46e8_isr_1125541_28.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/194df398-5cff-484f-be9c-cc1fe89b46e8_isr_1125541_28.stderr parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 91 submitted for App characterizeImage, waiting on task 90 parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 91 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 92 submitted for App calibrate, waiting on task 91 parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 92 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 93 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 93 has memoization hash 2132d2c39f98e8409dee94062c66f7d0 parsl.dataflow.memoization INFO: Task 93 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b74c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 32 parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/d2170479-9ad1-4e76-988b-3ed99dbe258b_isr_1125541_37.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/d2170479-9ad1-4e76-988b-3ed99dbe258b_isr_1125541_37.stderr parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 94 submitted for App characterizeImage, waiting on task 93 parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 94 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 95 submitted for App calibrate, waiting on task 94 parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 95 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 96 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 96 has memoization hash fd56d6002b812a6ef78eb653d5d732ba parsl.dataflow.memoization INFO: Task 96 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b77e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 33 parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/2cf93e14-fdf8-4bbc-90b3-ad3eaf024386_isr_1125543_39.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/2cf93e14-fdf8-4bbc-90b3-ad3eaf024386_isr_1125543_39.stderr parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 97 submitted for App characterizeImage, waiting on task 96 parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 97 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 98 submitted for App calibrate, waiting on task 97 parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 98 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 99 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 99 has memoization hash 9ee7445941a24292cf7f6032765ce914 parsl.dataflow.memoization INFO: Task 99 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b7b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 34 parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/7d625fe9-1d77-43dc-87d5-bbbc66a9928b_isr_1125541_6.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/7d625fe9-1d77-43dc-87d5-bbbc66a9928b_isr_1125541_6.stderr parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 100 submitted for App characterizeImage, waiting on task 99 parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 100 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 101 submitted for App calibrate, waiting on task 100 parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 101 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 102 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 102 has memoization hash 585ac54d2157788f9d4f9ec818edfc16 parsl.dataflow.memoization INFO: Task 102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b7e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 35 parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/1ceab569-76b6-4172-9787-949b7c73d5f3_isr_1125543_27.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/1ceab569-76b6-4172-9787-949b7c73d5f3_isr_1125543_27.stderr parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 103 submitted for App characterizeImage, waiting on task 102 parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 103 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 104 submitted for App calibrate, waiting on task 103 parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 104 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 105 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 105 has memoization hash 1e9925691120c8993d704505c5151d51 parsl.dataflow.memoization INFO: Task 105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eac180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 36 parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/fc8ee954-7a5e-4065-976c-233f536aae41_isr_1125541_62.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/fc8ee954-7a5e-4065-976c-233f536aae41_isr_1125541_62.stderr parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 106 submitted for App characterizeImage, waiting on task 105 parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 106 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 107 submitted for App calibrate, waiting on task 106 parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 107 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 108 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 108 has memoization hash 8242c56d7220057fa1179cf94e558c4d parsl.dataflow.memoization INFO: Task 108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eac4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 37 parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/b4f10e75-540c-4814-8ad1-038a0a5e48e5_isr_1125542_19.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/b4f10e75-540c-4814-8ad1-038a0a5e48e5_isr_1125542_19.stderr parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 109 submitted for App characterizeImage, waiting on task 108 parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 109 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 110 submitted for App calibrate, waiting on task 109 parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 110 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 111 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 111 has memoization hash dc58928f786d36ac44bfacfe69e217f3 parsl.dataflow.memoization INFO: Task 111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eac7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 38 parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/1a435449-6d93-45fb-9003-c724f82559e7_isr_1125543_20.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/1a435449-6d93-45fb-9003-c724f82559e7_isr_1125543_20.stderr parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 112 submitted for App characterizeImage, waiting on task 111 parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 112 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 113 submitted for App calibrate, waiting on task 112 parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 113 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 114 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 114 has memoization hash 40173fb64cf55cd6f93ccef7bc0b741c parsl.dataflow.memoization INFO: Task 114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eacae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 39 parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/748bf058-9379-48bc-8549-2cb31fedebe6_isr_1125542_20.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/748bf058-9379-48bc-8549-2cb31fedebe6_isr_1125542_20.stderr parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 115 submitted for App characterizeImage, waiting on task 114 parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 115 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 116 submitted for App calibrate, waiting on task 115 parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 116 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 117 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 117 has memoization hash 5291d3578a6965cf62f35478044cff0f parsl.dataflow.memoization INFO: Task 117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eace00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 40 parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/d9d09485-d908-4a38-8605-86adcb02a7fe_isr_1125542_52.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/d9d09485-d908-4a38-8605-86adcb02a7fe_isr_1125542_52.stderr parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 118 submitted for App characterizeImage, waiting on task 117 parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 118 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 119 submitted for App calibrate, waiting on task 118 parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 119 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 120 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 120 has memoization hash 4c61ccd1a78099b20d016a907ec894d8 parsl.dataflow.memoization INFO: Task 120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ead120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 41 parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/a18f6aeb-43b0-4eda-8e40-71ff6d1b1303_isr_1125542_2.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/a18f6aeb-43b0-4eda-8e40-71ff6d1b1303_isr_1125542_2.stderr parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 121 submitted for App characterizeImage, waiting on task 120 parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 121 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 122 submitted for App calibrate, waiting on task 121 parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 122 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 123 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 123 has memoization hash 908b111b0641fe5168726cc3726c04e5 parsl.dataflow.memoization INFO: Task 123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ead440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 42 parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/23592d35-155b-4100-9d1d-aabd867391f4_isr_1125541_21.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/23592d35-155b-4100-9d1d-aabd867391f4_isr_1125541_21.stderr parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 124 submitted for App characterizeImage, waiting on task 123 parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 124 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 125 submitted for App calibrate, waiting on task 124 parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 125 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 126 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 126 has memoization hash 210b68d2e224af75cf0e9b760e3e12df parsl.dataflow.memoization INFO: Task 126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ead760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 43 parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/920f8a24-ac55-4822-aa99-629f7a2270d1_isr_1125542_43.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/920f8a24-ac55-4822-aa99-629f7a2270d1_isr_1125542_43.stderr parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 127 submitted for App characterizeImage, waiting on task 126 parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 127 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 128 submitted for App calibrate, waiting on task 127 parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 128 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 129 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 129 has memoization hash 71cb68c73f22e247d7048c05c04eb464 parsl.dataflow.memoization INFO: Task 129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eada80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 129 try 0 launched on executor multi with executor id 44 parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/c1f5c588-c02f-473a-94bc-4390c02fa006_isr_1125542_4.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/c1f5c588-c02f-473a-94bc-4390c02fa006_isr_1125542_4.stderr parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 130 submitted for App characterizeImage, waiting on task 129 parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 130 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 131 submitted for App calibrate, waiting on task 130 parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 131 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 132 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 132 has memoization hash 30abbeedeccd2e6dd451353c07d5ae1f parsl.dataflow.memoization INFO: Task 132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eadda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 45 parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/dec91c94-6ab0-4f82-b99a-01a284b775c9_isr_1125543_13.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/dec91c94-6ab0-4f82-b99a-01a284b775c9_isr_1125543_13.stderr parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 133 submitted for App characterizeImage, waiting on task 132 parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 133 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 134 submitted for App calibrate, waiting on task 133 parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 134 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 135 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 135 has memoization hash a110cbe9496475cc87eab3d9b86f146c parsl.dataflow.memoization INFO: Task 135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eae0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 135 try 0 launched on executor multi with executor id 46 parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/124b3a74-3f98-4016-abd8-f4db9a10797d_isr_1125543_24.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/124b3a74-3f98-4016-abd8-f4db9a10797d_isr_1125543_24.stderr parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 136 submitted for App characterizeImage, waiting on task 135 parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 136 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 137 submitted for App calibrate, waiting on task 136 parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 137 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 138 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 138 has memoization hash 9055ba5a4cad9910e2c72350a68f0046 parsl.dataflow.memoization INFO: Task 138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eae3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 47 parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/ca8aa607-12f0-469b-8322-612680752d37_isr_1125543_9.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/ca8aa607-12f0-469b-8322-612680752d37_isr_1125543_9.stderr parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 139 submitted for App characterizeImage, waiting on task 138 parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 139 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 140 submitted for App calibrate, waiting on task 139 parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 140 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 141 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 141 has memoization hash 506674e2f66933b63c8ceb9b2ae8d047 parsl.dataflow.memoization INFO: Task 141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eae700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 141 try 0 launched on executor multi with executor id 48 parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/17d19bf6-dfe4-40be-9520-35abe62c6c62_isr_1125543_18.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/17d19bf6-dfe4-40be-9520-35abe62c6c62_isr_1125543_18.stderr parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 142 submitted for App characterizeImage, waiting on task 141 parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 142 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 143 submitted for App calibrate, waiting on task 142 parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 143 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 144 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 144 has memoization hash 050e2ee3143895db2b66bbba1c274622 parsl.dataflow.memoization INFO: Task 144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eaea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 49 parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/fc6fdf3e-d296-4734-8973-2587c8f3c87a_isr_1125541_42.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/fc6fdf3e-d296-4734-8973-2587c8f3c87a_isr_1125541_42.stderr parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 145 submitted for App characterizeImage, waiting on task 144 parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 145 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 146 submitted for App calibrate, waiting on task 145 parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 146 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 147 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 147 has memoization hash 17e2cf50cb89b61b3d090487f459289a parsl.dataflow.memoization INFO: Task 147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eaed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 147 try 0 launched on executor multi with executor id 50 parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/123f7c9c-ae73-4f00-a26b-d6970d633256_isr_1125542_31.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/123f7c9c-ae73-4f00-a26b-d6970d633256_isr_1125542_31.stderr parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 148 submitted for App characterizeImage, waiting on task 147 parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 148 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 149 submitted for App calibrate, waiting on task 148 parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 149 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 150 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 150 has memoization hash 2762660298cb9f5006a91010dbe2ed56 parsl.dataflow.memoization INFO: Task 150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eaf060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 51 parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/033922a1-7260-4f58-ba52-ceb4ac82ef9c_isr_1125542_7.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/033922a1-7260-4f58-ba52-ceb4ac82ef9c_isr_1125542_7.stderr parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 151 submitted for App characterizeImage, waiting on task 150 parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 151 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 152 submitted for App calibrate, waiting on task 151 parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 152 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 153 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 153 has memoization hash c22425bc04d62e66441c0bb5f5bd5c5f parsl.dataflow.memoization INFO: Task 153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eaf380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 153 try 0 launched on executor multi with executor id 52 parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/982ea051-4f70-4582-95e5-959594edae2e_isr_1125543_41.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/982ea051-4f70-4582-95e5-959594edae2e_isr_1125543_41.stderr parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 154 submitted for App characterizeImage, waiting on task 153 parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 154 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 155 submitted for App calibrate, waiting on task 154 parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 155 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 156 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 156 has memoization hash 7fc0b841403d231f317405b431261f67 parsl.dataflow.memoization INFO: Task 156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eaf6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 53 parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/ff807c24-b95b-4207-895b-116229c29a0e_isr_1125542_15.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/ff807c24-b95b-4207-895b-116229c29a0e_isr_1125542_15.stderr parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 157 submitted for App characterizeImage, waiting on task 156 parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 157 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 158 submitted for App calibrate, waiting on task 157 parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 158 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 159 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 159 has memoization hash 104ed21ae6416a637b5bc87e53099949 parsl.dataflow.memoization INFO: Task 159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eaf9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 159 try 0 launched on executor multi with executor id 54 parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/4529f0f8-22b8-4cb2-84a5-417ac8c724f0_isr_1125542_61.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/4529f0f8-22b8-4cb2-84a5-417ac8c724f0_isr_1125542_61.stderr parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 160 submitted for App characterizeImage, waiting on task 159 parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 160 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 161 submitted for App calibrate, waiting on task 160 parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 161 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 162 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 162 has memoization hash 49f59f5ae646842ba8ca216c679a96d8 parsl.dataflow.memoization INFO: Task 162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eafce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 55 parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/ed349b7f-4d20-4b6e-8209-fd770f7aaeed_isr_1125542_35.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/ed349b7f-4d20-4b6e-8209-fd770f7aaeed_isr_1125542_35.stderr parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 163 submitted for App characterizeImage, waiting on task 162 parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 163 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 164 submitted for App calibrate, waiting on task 163 parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 164 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 165 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 165 has memoization hash af301d8b1244fef82dea95746f9c257b parsl.dataflow.memoization INFO: Task 165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed4040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 165 try 0 launched on executor multi with executor id 56 parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/a61b60f1-0e7c-406d-9e55-ddd55578d708_isr_1125543_11.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/a61b60f1-0e7c-406d-9e55-ddd55578d708_isr_1125543_11.stderr parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 166 submitted for App characterizeImage, waiting on task 165 parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 166 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 167 submitted for App calibrate, waiting on task 166 parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 167 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 168 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 168 has memoization hash b8e9d245e254198bf317e35cf4919c58 parsl.dataflow.memoization INFO: Task 168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed4360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 57 parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/53b08f40-6442-4d07-bbd9-a80bde0ec250_isr_1125541_51.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/53b08f40-6442-4d07-bbd9-a80bde0ec250_isr_1125541_51.stderr parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 169 submitted for App characterizeImage, waiting on task 168 parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 169 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 170 submitted for App calibrate, waiting on task 169 parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 170 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 171 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 171 has memoization hash f06b4f3dd1c945f08458b9bc46e9a493 parsl.dataflow.memoization INFO: Task 171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed4680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 171 try 0 launched on executor multi with executor id 58 parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/eb757be7-724d-425a-83e1-fff1ad34c7a1_isr_1125541_35.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/eb757be7-724d-425a-83e1-fff1ad34c7a1_isr_1125541_35.stderr parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 172 submitted for App characterizeImage, waiting on task 171 parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 172 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 173 submitted for App calibrate, waiting on task 172 parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 173 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 174 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 174 has memoization hash 5260187c5f0c3a60d05e2432e9baac21 parsl.dataflow.memoization INFO: Task 174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed49a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 59 parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/d2c6ea6d-9137-4787-9ecc-d6cd4f043643_isr_1125543_6.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/d2c6ea6d-9137-4787-9ecc-d6cd4f043643_isr_1125543_6.stderr parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 175 submitted for App characterizeImage, waiting on task 174 parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 175 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 176 submitted for App calibrate, waiting on task 175 parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 176 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 177 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 177 has memoization hash 66294f4cd5fa22b21c02fbf896337e28 parsl.dataflow.memoization INFO: Task 177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed4cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 177 try 0 launched on executor multi with executor id 60 parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/6b1c30da-1bf8-4f33-bcda-5f08991500fd_isr_1125542_10.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/6b1c30da-1bf8-4f33-bcda-5f08991500fd_isr_1125542_10.stderr parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 178 submitted for App characterizeImage, waiting on task 177 parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 178 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 179 submitted for App calibrate, waiting on task 178 parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 179 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 180 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 180 has memoization hash 0f90321872e46e43c54e0443d5252b65 parsl.dataflow.memoization INFO: Task 180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed4fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 61 parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/43aee40d-5a1c-4747-ad4c-2cd9de70c068_isr_1125542_42.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/43aee40d-5a1c-4747-ad4c-2cd9de70c068_isr_1125542_42.stderr parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 181 submitted for App characterizeImage, waiting on task 180 parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 181 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 182 submitted for App calibrate, waiting on task 181 parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 182 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 183 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 183 has memoization hash 0715d7a6d1edacca927a075f046d0675 parsl.dataflow.memoization INFO: Task 183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed5300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 183 try 0 launched on executor multi with executor id 62 parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/2fdd7122-daf9-4988-9333-727c3c57166d_isr_1125543_59.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/2fdd7122-daf9-4988-9333-727c3c57166d_isr_1125543_59.stderr parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 184 submitted for App characterizeImage, waiting on task 183 parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 184 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 185 submitted for App calibrate, waiting on task 184 parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 185 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 186 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 186 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 186 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 186 has memoization hash 88105c22d5608a1b56bdf66e038f2cc3 parsl.dataflow.memoization INFO: Task 186 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed5620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 186 try 0 launched on executor multi with executor id 63 parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/51b1c61b-2b93-4c3b-b26b-3af364d820de_isr_1125541_43.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/51b1c61b-2b93-4c3b-b26b-3af364d820de_isr_1125541_43.stderr parsl.dataflow.dflow DEBUG: Task 187 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 187 submitted for App characterizeImage, waiting on task 186 parsl.dataflow.dflow DEBUG: Task 187 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 187 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 188 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 188 submitted for App calibrate, waiting on task 187 parsl.dataflow.dflow DEBUG: Task 188 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 188 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 189 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 189 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 189 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 189 has memoization hash 76cc6f323a78c2dbd1bc6e3b614a4a37 parsl.dataflow.memoization INFO: Task 189 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed5940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 189 try 0 launched on executor multi with executor id 64 parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/4610b7ea-9c87-4a78-8ba2-0854f64d7f05_isr_1125543_57.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/4610b7ea-9c87-4a78-8ba2-0854f64d7f05_isr_1125543_57.stderr parsl.dataflow.dflow DEBUG: Task 190 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 190 submitted for App characterizeImage, waiting on task 189 parsl.dataflow.dflow DEBUG: Task 190 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 190 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 191 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 191 submitted for App calibrate, waiting on task 190 parsl.dataflow.dflow DEBUG: Task 191 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 191 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 192 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 192 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 192 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 192 has memoization hash f673d2c72298e4bef169dd644b035230 parsl.dataflow.memoization INFO: Task 192 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed5c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 192 try 0 launched on executor multi with executor id 65 parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/8a35b649-ce28-425f-98d6-7237bfd18fbd_isr_1125542_13.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/8a35b649-ce28-425f-98d6-7237bfd18fbd_isr_1125542_13.stderr parsl.dataflow.dflow DEBUG: Task 193 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 193 submitted for App characterizeImage, waiting on task 192 parsl.dataflow.dflow DEBUG: Task 193 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 193 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 194 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 194 submitted for App calibrate, waiting on task 193 parsl.dataflow.dflow DEBUG: Task 194 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 194 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 195 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 195 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 195 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 195 has memoization hash 99268a62769be421707305bf42826054 parsl.dataflow.memoization INFO: Task 195 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed5f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 195 try 0 launched on executor multi with executor id 66 parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/cea4f72d-3bd0-4851-854c-7153bfa3781e_isr_1125541_61.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/cea4f72d-3bd0-4851-854c-7153bfa3781e_isr_1125541_61.stderr parsl.dataflow.dflow DEBUG: Task 196 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 196 submitted for App characterizeImage, waiting on task 195 parsl.dataflow.dflow DEBUG: Task 196 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 196 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 197 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 197 submitted for App calibrate, waiting on task 196 parsl.dataflow.dflow DEBUG: Task 197 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 197 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 198 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 198 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 198 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 198 has memoization hash 5e459c19e4b29343208a88f1737f9738 parsl.dataflow.memoization INFO: Task 198 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed62a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 198 try 0 launched on executor multi with executor id 67 parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/f02bea9c-0862-441d-ac50-1577f83909b7_isr_1125543_43.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/f02bea9c-0862-441d-ac50-1577f83909b7_isr_1125543_43.stderr parsl.dataflow.dflow DEBUG: Task 199 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 199 submitted for App characterizeImage, waiting on task 198 parsl.dataflow.dflow DEBUG: Task 199 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 199 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 200 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 200 submitted for App calibrate, waiting on task 199 parsl.dataflow.dflow DEBUG: Task 200 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 200 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 201 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 201 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 201 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 201 has memoization hash 87f82dfd235440dc3247d0be04a56ba3 parsl.dataflow.memoization INFO: Task 201 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed65c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 201 try 0 launched on executor multi with executor id 68 parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/8547c195-a175-419e-8619-0df565950dfa_isr_1125543_56.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/8547c195-a175-419e-8619-0df565950dfa_isr_1125543_56.stderr parsl.dataflow.dflow DEBUG: Task 202 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 202 submitted for App characterizeImage, waiting on task 201 parsl.dataflow.dflow DEBUG: Task 202 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 202 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 203 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 203 submitted for App calibrate, waiting on task 202 parsl.dataflow.dflow DEBUG: Task 203 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 203 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 204 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 204 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 204 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 204 has memoization hash b0905b0904b20e27827b038d048ef001 parsl.dataflow.memoization INFO: Task 204 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed68e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 204 try 0 launched on executor multi with executor id 69 parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/2ad81a7b-e981-4d09-9181-ee2e21fc8abf_isr_1125542_51.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/2ad81a7b-e981-4d09-9181-ee2e21fc8abf_isr_1125542_51.stderr parsl.dataflow.dflow DEBUG: Task 205 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 205 submitted for App characterizeImage, waiting on task 204 parsl.dataflow.dflow DEBUG: Task 205 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 205 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 206 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 206 submitted for App calibrate, waiting on task 205 parsl.dataflow.dflow DEBUG: Task 206 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 206 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 207 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 207 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 207 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 207 has memoization hash 07796be154ad47bfa0aced66509b1098 parsl.dataflow.memoization INFO: Task 207 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed6c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 207 try 0 launched on executor multi with executor id 70 parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/31bf069b-606a-44a6-a5f2-92b81bb4b5dc_isr_1125541_48.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/31bf069b-606a-44a6-a5f2-92b81bb4b5dc_isr_1125541_48.stderr parsl.dataflow.dflow DEBUG: Task 208 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 208 submitted for App characterizeImage, waiting on task 207 parsl.dataflow.dflow DEBUG: Task 208 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 208 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 209 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 209 submitted for App calibrate, waiting on task 208 parsl.dataflow.dflow DEBUG: Task 209 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 209 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 210 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 210 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 210 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 210 has memoization hash 232fb851d88ab91389f0e930ae992f76 parsl.dataflow.memoization INFO: Task 210 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed6f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 210 try 0 launched on executor multi with executor id 71 parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/1d49f597-ab8c-4bc2-8a5d-ce82072fe515_isr_1125542_29.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/1d49f597-ab8c-4bc2-8a5d-ce82072fe515_isr_1125542_29.stderr parsl.dataflow.dflow DEBUG: Task 211 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 211 submitted for App characterizeImage, waiting on task 210 parsl.dataflow.dflow DEBUG: Task 211 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 211 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 212 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 212 submitted for App calibrate, waiting on task 211 parsl.dataflow.dflow DEBUG: Task 212 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 212 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 213 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 213 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 213 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 213 has memoization hash 43a03558eab2096d28b6acc0861dc080 parsl.dataflow.memoization INFO: Task 213 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed7240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 213 try 0 launched on executor multi with executor id 72 parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/6319ecc3-222f-47de-8f29-c9fce168ec24_isr_1125542_48.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/6319ecc3-222f-47de-8f29-c9fce168ec24_isr_1125542_48.stderr parsl.dataflow.dflow DEBUG: Task 214 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 214 submitted for App characterizeImage, waiting on task 213 parsl.dataflow.dflow DEBUG: Task 214 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 214 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 215 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 215 submitted for App calibrate, waiting on task 214 parsl.dataflow.dflow DEBUG: Task 215 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 215 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 216 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 216 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 216 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 216 has memoization hash 3cb09fba5a7a3fee5aa28132b168626f parsl.dataflow.memoization INFO: Task 216 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed7560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 216 try 0 launched on executor multi with executor id 73 parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/7339672c-ae9e-4602-b7e3-319bafcd1b0a_isr_1125542_32.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/7339672c-ae9e-4602-b7e3-319bafcd1b0a_isr_1125542_32.stderr parsl.dataflow.dflow DEBUG: Task 217 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 217 submitted for App characterizeImage, waiting on task 216 parsl.dataflow.dflow DEBUG: Task 217 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 217 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 218 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 218 submitted for App calibrate, waiting on task 217 parsl.dataflow.dflow DEBUG: Task 218 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 218 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 219 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 219 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 219 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 219 has memoization hash 7e76ea1df35edd22b91bdac87337d8f9 parsl.dataflow.memoization INFO: Task 219 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed7880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 219 try 0 launched on executor multi with executor id 74 parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/2c3a7240-e51f-4d2d-b970-dc75a49c81cb_isr_1125542_23.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/2c3a7240-e51f-4d2d-b970-dc75a49c81cb_isr_1125542_23.stderr parsl.dataflow.dflow DEBUG: Task 220 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 220 submitted for App characterizeImage, waiting on task 219 parsl.dataflow.dflow DEBUG: Task 220 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 220 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 221 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 221 submitted for App calibrate, waiting on task 220 parsl.dataflow.dflow DEBUG: Task 221 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 221 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 222 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 222 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 222 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 222 has memoization hash 1b3dbe0aeed1b168fd03e84ddbbc4b9b parsl.dataflow.memoization INFO: Task 222 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed7ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 222 try 0 launched on executor multi with executor id 75 parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/31ec9599-2a0b-4056-b619-ab8af8a4c089_isr_1125543_48.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/31ec9599-2a0b-4056-b619-ab8af8a4c089_isr_1125543_48.stderr parsl.dataflow.dflow DEBUG: Task 223 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 223 submitted for App characterizeImage, waiting on task 222 parsl.dataflow.dflow DEBUG: Task 223 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 223 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 224 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 224 submitted for App calibrate, waiting on task 223 parsl.dataflow.dflow DEBUG: Task 224 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 224 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 225 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 225 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 225 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 225 has memoization hash 694adb729cfc301813567e24677b3107 parsl.dataflow.memoization INFO: Task 225 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed7ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 225 try 0 launched on executor multi with executor id 76 parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/6e74b57b-a936-458c-acd6-78115344feee_isr_1125543_51.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/6e74b57b-a936-458c-acd6-78115344feee_isr_1125543_51.stderr parsl.dataflow.dflow DEBUG: Task 226 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 226 submitted for App characterizeImage, waiting on task 225 parsl.dataflow.dflow DEBUG: Task 226 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 226 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 227 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 227 submitted for App calibrate, waiting on task 226 parsl.dataflow.dflow DEBUG: Task 227 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 227 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 228 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 228 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 228 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 228 has memoization hash b6f1406e2fe5422ce5ce43d76a45224c parsl.dataflow.memoization INFO: Task 228 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efc220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 228 try 0 launched on executor multi with executor id 77 parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/2f5add86-0d46-4e7d-b629-43b7f7f30f7e_isr_1125542_12.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/2f5add86-0d46-4e7d-b629-43b7f7f30f7e_isr_1125542_12.stderr parsl.dataflow.dflow DEBUG: Task 229 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 229 submitted for App characterizeImage, waiting on task 228 parsl.dataflow.dflow DEBUG: Task 229 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 229 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 230 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 230 submitted for App calibrate, waiting on task 229 parsl.dataflow.dflow DEBUG: Task 230 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 230 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 231 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 231 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 231 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 231 has memoization hash c911b3d5cb40ab1075286576edc73d56 parsl.dataflow.memoization INFO: Task 231 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efc540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 231 try 0 launched on executor multi with executor id 78 parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/05eb6f32-ed8a-430c-92f0-c355a7a62b28_isr_1125543_14.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/05eb6f32-ed8a-430c-92f0-c355a7a62b28_isr_1125543_14.stderr parsl.dataflow.dflow DEBUG: Task 232 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 232 submitted for App characterizeImage, waiting on task 231 parsl.dataflow.dflow DEBUG: Task 232 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 232 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 233 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 233 submitted for App calibrate, waiting on task 232 parsl.dataflow.dflow DEBUG: Task 233 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 233 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 234 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 234 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 234 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 234 has memoization hash b86b17fd58ebf6e8cbba44e8241eff1b parsl.dataflow.memoization INFO: Task 234 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efc860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 234 try 0 launched on executor multi with executor id 79 parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/4365a311-8994-4e72-9dca-10d507522feb_isr_1125543_34.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/4365a311-8994-4e72-9dca-10d507522feb_isr_1125543_34.stderr parsl.dataflow.dflow DEBUG: Task 235 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 235 submitted for App characterizeImage, waiting on task 234 parsl.dataflow.dflow DEBUG: Task 235 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 235 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 236 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 236 submitted for App calibrate, waiting on task 235 parsl.dataflow.dflow DEBUG: Task 236 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 236 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 237 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 237 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 237 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 237 has memoization hash 0472ecb74c3b4f072861b0a9f8892b5a parsl.dataflow.memoization INFO: Task 237 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efcb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 237 try 0 launched on executor multi with executor id 80 parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/effe8e41-50b4-4769-a9ad-4102c756300a_isr_1125543_4.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/effe8e41-50b4-4769-a9ad-4102c756300a_isr_1125543_4.stderr parsl.dataflow.dflow DEBUG: Task 238 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 238 submitted for App characterizeImage, waiting on task 237 parsl.dataflow.dflow DEBUG: Task 238 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 238 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 239 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 239 submitted for App calibrate, waiting on task 238 parsl.dataflow.dflow DEBUG: Task 239 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 239 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 240 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 240 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 240 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 240 has memoization hash 861654228e9bdd58c7f6ee7a629654d7 parsl.dataflow.memoization INFO: Task 240 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efcea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 240 try 0 launched on executor multi with executor id 81 parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/2b3ebb3c-9295-4f1b-a2c3-5ce98a6b1aae_isr_1125541_30.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/2b3ebb3c-9295-4f1b-a2c3-5ce98a6b1aae_isr_1125541_30.stderr parsl.dataflow.dflow DEBUG: Task 241 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 241 submitted for App characterizeImage, waiting on task 240 parsl.dataflow.dflow DEBUG: Task 241 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 241 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 242 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 242 submitted for App calibrate, waiting on task 241 parsl.dataflow.dflow DEBUG: Task 242 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 242 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 243 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 243 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 243 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 243 has memoization hash 1624f367bb419409697a2347d935dfc0 parsl.dataflow.memoization INFO: Task 243 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efd1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 243 try 0 launched on executor multi with executor id 82 parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/eadfbc93-ac9b-4759-88a2-51176d7b8090_isr_1125542_6.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/eadfbc93-ac9b-4759-88a2-51176d7b8090_isr_1125542_6.stderr parsl.dataflow.dflow DEBUG: Task 244 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 244 submitted for App characterizeImage, waiting on task 243 parsl.dataflow.dflow DEBUG: Task 244 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 244 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 245 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 245 submitted for App calibrate, waiting on task 244 parsl.dataflow.dflow DEBUG: Task 245 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 245 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 246 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 246 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 246 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 246 has memoization hash 1f4ade892f05c2efb1363e0c2c88bcae parsl.dataflow.memoization INFO: Task 246 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efd4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 246 try 0 launched on executor multi with executor id 83 parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/33fa5b81-ba68-4e2e-bd1e-5b575381b232_isr_1125542_28.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/33fa5b81-ba68-4e2e-bd1e-5b575381b232_isr_1125542_28.stderr parsl.dataflow.dflow DEBUG: Task 247 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 247 submitted for App characterizeImage, waiting on task 246 parsl.dataflow.dflow DEBUG: Task 247 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 247 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 248 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 248 submitted for App calibrate, waiting on task 247 parsl.dataflow.dflow DEBUG: Task 248 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 248 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 249 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 249 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 249 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 249 has memoization hash f1108b4bc71c452a28cf16288002252c parsl.dataflow.memoization INFO: Task 249 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efd800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 249 try 0 launched on executor multi with executor id 84 parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/9956a44d-5ba5-45fa-afbd-d60fc47bcbbc_isr_1125542_49.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/9956a44d-5ba5-45fa-afbd-d60fc47bcbbc_isr_1125542_49.stderr parsl.dataflow.dflow DEBUG: Task 250 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 250 submitted for App characterizeImage, waiting on task 249 parsl.dataflow.dflow DEBUG: Task 250 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 250 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 251 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 251 submitted for App calibrate, waiting on task 250 parsl.dataflow.dflow DEBUG: Task 251 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 251 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 252 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 252 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 252 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 252 has memoization hash b5f3878731509ee804468a371fa9ce5f parsl.dataflow.memoization INFO: Task 252 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efdb20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 252 try 0 launched on executor multi with executor id 85 parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/f36a7d07-ae4c-4690-ab2f-955704e8cef7_isr_1125541_49.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/f36a7d07-ae4c-4690-ab2f-955704e8cef7_isr_1125541_49.stderr parsl.dataflow.dflow DEBUG: Task 253 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 253 submitted for App characterizeImage, waiting on task 252 parsl.dataflow.dflow DEBUG: Task 253 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 253 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 254 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 254 submitted for App calibrate, waiting on task 253 parsl.dataflow.dflow DEBUG: Task 254 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 254 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 255 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 255 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 255 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 255 has memoization hash feb248cbb06441ff3fea5261a68b7320 parsl.dataflow.memoization INFO: Task 255 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efde40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 255 try 0 launched on executor multi with executor id 86 parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/ed193144-db1d-4f6b-b064-e4216243c022_isr_1125542_9.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/ed193144-db1d-4f6b-b064-e4216243c022_isr_1125542_9.stderr parsl.dataflow.dflow DEBUG: Task 256 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 256 submitted for App characterizeImage, waiting on task 255 parsl.dataflow.dflow DEBUG: Task 256 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 256 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 257 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 257 submitted for App calibrate, waiting on task 256 parsl.dataflow.dflow DEBUG: Task 257 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 257 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 258 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 258 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 258 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 258 has memoization hash c87a71439dc8649cf1bb7bdb57a628b6 parsl.dataflow.memoization INFO: Task 258 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efe160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 258 try 0 launched on executor multi with executor id 87 parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/00dee20c-ce45-4778-a2a5-81ac0b8e2ffc_isr_1125543_58.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/00dee20c-ce45-4778-a2a5-81ac0b8e2ffc_isr_1125543_58.stderr parsl.dataflow.dflow DEBUG: Task 259 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 259 submitted for App characterizeImage, waiting on task 258 parsl.dataflow.dflow DEBUG: Task 259 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 259 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 260 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 260 submitted for App calibrate, waiting on task 259 parsl.dataflow.dflow DEBUG: Task 260 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 260 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 261 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 261 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 261 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 261 has memoization hash 373cc681d56afb76d78803e8ed0c69e2 parsl.dataflow.memoization INFO: Task 261 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efe480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 261 try 0 launched on executor multi with executor id 88 parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/89ae558c-7a3a-4f19-8ae5-310185ee02e4_isr_1125541_9.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/89ae558c-7a3a-4f19-8ae5-310185ee02e4_isr_1125541_9.stderr parsl.dataflow.dflow DEBUG: Task 262 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 262 submitted for App characterizeImage, waiting on task 261 parsl.dataflow.dflow DEBUG: Task 262 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 262 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 263 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 263 submitted for App calibrate, waiting on task 262 parsl.dataflow.dflow DEBUG: Task 263 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 263 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 264 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 264 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 264 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 264 has memoization hash eb1dfad588d839aeb513c408ae3b00bb parsl.dataflow.memoization INFO: Task 264 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efe7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 264 try 0 launched on executor multi with executor id 89 parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/cfce0014-14bd-4e3f-994f-2774c53561cf_isr_1125542_17.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/cfce0014-14bd-4e3f-994f-2774c53561cf_isr_1125542_17.stderr parsl.dataflow.dflow DEBUG: Task 265 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 265 submitted for App characterizeImage, waiting on task 264 parsl.dataflow.dflow DEBUG: Task 265 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 265 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 266 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 266 submitted for App calibrate, waiting on task 265 parsl.dataflow.dflow DEBUG: Task 266 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 266 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 267 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 267 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 267 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 267 has memoization hash 53dd1a0ecd45db616a1b1dcd4e003bf0 parsl.dataflow.memoization INFO: Task 267 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efeac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 267 try 0 launched on executor multi with executor id 90 parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/6d3bd26f-7a88-4440-b165-be27005649f5_isr_1125542_25.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/6d3bd26f-7a88-4440-b165-be27005649f5_isr_1125542_25.stderr parsl.dataflow.dflow DEBUG: Task 268 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 268 submitted for App characterizeImage, waiting on task 267 parsl.dataflow.dflow DEBUG: Task 268 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 268 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 269 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 269 submitted for App calibrate, waiting on task 268 parsl.dataflow.dflow DEBUG: Task 269 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 269 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 270 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 270 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 270 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 270 has memoization hash f7243399de7dec313e3140db43e96676 parsl.dataflow.memoization INFO: Task 270 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 270 try 0 launched on executor multi with executor id 91 parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/56b79f35-328d-4e06-9bbd-b180653f4bd0_isr_1125543_36.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/56b79f35-328d-4e06-9bbd-b180653f4bd0_isr_1125543_36.stderr parsl.dataflow.dflow DEBUG: Task 271 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 271 submitted for App characterizeImage, waiting on task 270 parsl.dataflow.dflow DEBUG: Task 271 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 271 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 272 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 272 submitted for App calibrate, waiting on task 271 parsl.dataflow.dflow DEBUG: Task 272 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 272 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 273 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 273 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 273 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 273 has memoization hash dace0a55e7fea54e1621583f9848cbf6 parsl.dataflow.memoization INFO: Task 273 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eff100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 273 try 0 launched on executor multi with executor id 92 parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/be4652cc-b5cb-4127-9a29-d67ab4f0f2be_isr_1125541_24.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/be4652cc-b5cb-4127-9a29-d67ab4f0f2be_isr_1125541_24.stderr parsl.dataflow.dflow DEBUG: Task 274 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 274 submitted for App characterizeImage, waiting on task 273 parsl.dataflow.dflow DEBUG: Task 274 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 274 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 275 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 275 submitted for App calibrate, waiting on task 274 parsl.dataflow.dflow DEBUG: Task 275 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 275 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 276 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 276 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 276 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 276 has memoization hash ab3647035755ee6a84801aac26ce1901 parsl.dataflow.memoization INFO: Task 276 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eff420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 276 try 0 launched on executor multi with executor id 93 parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/6d3455f2-4cff-4df7-920f-7799f3b39c5d_isr_1125541_1.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/6d3455f2-4cff-4df7-920f-7799f3b39c5d_isr_1125541_1.stderr parsl.dataflow.dflow DEBUG: Task 277 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 277 submitted for App characterizeImage, waiting on task 276 parsl.dataflow.dflow DEBUG: Task 277 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 277 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 278 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 278 submitted for App calibrate, waiting on task 277 parsl.dataflow.dflow DEBUG: Task 278 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 278 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 279 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 279 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 279 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 279 has memoization hash 7e08803902ee658035c90cd5af303166 parsl.dataflow.memoization INFO: Task 279 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eff740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 279 try 0 launched on executor multi with executor id 94 parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/504a48d6-e1a9-45e1-be6e-d14a0e871d27_isr_1125543_54.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/504a48d6-e1a9-45e1-be6e-d14a0e871d27_isr_1125543_54.stderr parsl.dataflow.dflow DEBUG: Task 280 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 280 submitted for App characterizeImage, waiting on task 279 parsl.dataflow.dflow DEBUG: Task 280 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 280 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 281 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 281 submitted for App calibrate, waiting on task 280 parsl.dataflow.dflow DEBUG: Task 281 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 281 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 282 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 282 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 282 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 282 has memoization hash 0531573469ccddbd98dcdd5b5a244cdb parsl.dataflow.memoization INFO: Task 282 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811effa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 282 try 0 launched on executor multi with executor id 95 parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/94e3ef5a-e56f-467e-8fc0-2c7844cd7036_isr_1125543_7.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/94e3ef5a-e56f-467e-8fc0-2c7844cd7036_isr_1125543_7.stderr parsl.dataflow.dflow DEBUG: Task 283 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 283 submitted for App characterizeImage, waiting on task 282 parsl.dataflow.dflow DEBUG: Task 283 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 283 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 284 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 284 submitted for App calibrate, waiting on task 283 parsl.dataflow.dflow DEBUG: Task 284 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 284 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 285 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 285 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 285 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 285 has memoization hash e48e8be06cd0b01747b9c397541e1064 parsl.dataflow.memoization INFO: Task 285 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811effd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 285 try 0 launched on executor multi with executor id 96 parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/e55779fa-4853-4a1b-a83d-21bc25aa7343_isr_1125543_17.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/e55779fa-4853-4a1b-a83d-21bc25aa7343_isr_1125543_17.stderr parsl.dataflow.dflow DEBUG: Task 286 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 286 submitted for App characterizeImage, waiting on task 285 parsl.dataflow.dflow DEBUG: Task 286 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 286 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 287 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 287 submitted for App calibrate, waiting on task 286 parsl.dataflow.dflow DEBUG: Task 287 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 287 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 288 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 288 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 288 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 288 has memoization hash f47be76591ca0a9d97f5777665131e4d parsl.dataflow.memoization INFO: Task 288 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f300e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 288 try 0 launched on executor multi with executor id 97 parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/5095dd05-8431-45db-a081-055cf0e8757f_isr_1125542_53.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/5095dd05-8431-45db-a081-055cf0e8757f_isr_1125542_53.stderr parsl.dataflow.dflow DEBUG: Task 289 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 289 submitted for App characterizeImage, waiting on task 288 parsl.dataflow.dflow DEBUG: Task 289 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 289 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 290 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 290 submitted for App calibrate, waiting on task 289 parsl.dataflow.dflow DEBUG: Task 290 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 290 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 291 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 291 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 291 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 291 has memoization hash 659cc1457560720d246976b12608e171 parsl.dataflow.memoization INFO: Task 291 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f30400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 291 try 0 launched on executor multi with executor id 98 parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/803c99b0-0605-4b2e-9cec-033a144132ed_isr_1125541_55.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/803c99b0-0605-4b2e-9cec-033a144132ed_isr_1125541_55.stderr parsl.dataflow.dflow DEBUG: Task 292 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 292 submitted for App characterizeImage, waiting on task 291 parsl.dataflow.dflow DEBUG: Task 292 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 292 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 293 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 293 submitted for App calibrate, waiting on task 292 parsl.dataflow.dflow DEBUG: Task 293 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 293 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 294 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 294 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 294 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 294 has memoization hash 9a80230efa790aead214b06c81106854 parsl.dataflow.memoization INFO: Task 294 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f30720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 294 try 0 launched on executor multi with executor id 99 parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/e6078452-361b-4297-863f-5e5a1fc61790_isr_1125541_14.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/e6078452-361b-4297-863f-5e5a1fc61790_isr_1125541_14.stderr parsl.dataflow.dflow DEBUG: Task 295 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 295 submitted for App characterizeImage, waiting on task 294 parsl.dataflow.dflow DEBUG: Task 295 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 295 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 296 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 296 submitted for App calibrate, waiting on task 295 parsl.dataflow.dflow DEBUG: Task 296 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 296 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 297 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 297 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 297 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 297 has memoization hash 62b0a1702a230b9b6725e1666a9e26cb parsl.dataflow.memoization INFO: Task 297 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f30a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 297 try 0 launched on executor multi with executor id 100 parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/e36b8eab-3b72-40f9-b459-58947a6f8c80_isr_1125543_31.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/e36b8eab-3b72-40f9-b459-58947a6f8c80_isr_1125543_31.stderr parsl.dataflow.dflow DEBUG: Task 298 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 298 submitted for App characterizeImage, waiting on task 297 parsl.dataflow.dflow DEBUG: Task 298 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 298 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 299 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 299 submitted for App calibrate, waiting on task 298 parsl.dataflow.dflow DEBUG: Task 299 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 299 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 300 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 300 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 300 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 300 has memoization hash a5c38299cac5cbc7416af00b0d5b7d5d parsl.dataflow.memoization INFO: Task 300 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f30d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 300 try 0 launched on executor multi with executor id 101 parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/e4a5a323-7a8f-44ac-8dac-b63ce12afeeb_isr_1125543_22.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/e4a5a323-7a8f-44ac-8dac-b63ce12afeeb_isr_1125543_22.stderr parsl.dataflow.dflow DEBUG: Task 301 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 301 submitted for App characterizeImage, waiting on task 300 parsl.dataflow.dflow DEBUG: Task 301 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 301 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 302 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 302 submitted for App calibrate, waiting on task 301 parsl.dataflow.dflow DEBUG: Task 302 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 302 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 303 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 303 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 303 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 303 has memoization hash 1ba8907b3828bbb741f75eef48853614 parsl.dataflow.memoization INFO: Task 303 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f31080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 303 try 0 launched on executor multi with executor id 102 parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/2088ae61-58e3-4593-b349-f43b34bdffef_isr_1125541_46.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/2088ae61-58e3-4593-b349-f43b34bdffef_isr_1125541_46.stderr parsl.dataflow.dflow DEBUG: Task 304 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 304 submitted for App characterizeImage, waiting on task 303 parsl.dataflow.dflow DEBUG: Task 304 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 304 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 305 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 305 submitted for App calibrate, waiting on task 304 parsl.dataflow.dflow DEBUG: Task 305 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 305 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 306 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 306 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 306 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 306 has memoization hash bd4c213c8515a3b663ba51f8e46ddbcb parsl.dataflow.memoization INFO: Task 306 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f313a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 306 try 0 launched on executor multi with executor id 103 parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/9d7488c3-4f16-4cc8-8bc6-972c454ffe2d_isr_1125542_1.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/9d7488c3-4f16-4cc8-8bc6-972c454ffe2d_isr_1125542_1.stderr parsl.dataflow.dflow DEBUG: Task 307 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 307 submitted for App characterizeImage, waiting on task 306 parsl.dataflow.dflow DEBUG: Task 307 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 307 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 308 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 308 submitted for App calibrate, waiting on task 307 parsl.dataflow.dflow DEBUG: Task 308 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 308 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 309 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 309 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 309 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 309 has memoization hash 4171bf776044ffb651461f11319f6725 parsl.dataflow.memoization INFO: Task 309 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f316c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 309 try 0 launched on executor multi with executor id 104 parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/fc720e8e-dc6f-40cf-ade5-91d655a1a5e1_isr_1125542_16.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/fc720e8e-dc6f-40cf-ade5-91d655a1a5e1_isr_1125542_16.stderr parsl.dataflow.dflow DEBUG: Task 310 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 310 submitted for App characterizeImage, waiting on task 309 parsl.dataflow.dflow DEBUG: Task 310 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 310 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 311 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 311 submitted for App calibrate, waiting on task 310 parsl.dataflow.dflow DEBUG: Task 311 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 311 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 312 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 312 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 312 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 312 has memoization hash b1303aa9ee350822a5c3eaa72f5a7ee6 parsl.dataflow.memoization INFO: Task 312 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f319e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 312 try 0 launched on executor multi with executor id 105 parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/6fce3ea1-456e-459d-ac3f-345d10e7cf9c_isr_1125541_17.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/6fce3ea1-456e-459d-ac3f-345d10e7cf9c_isr_1125541_17.stderr parsl.dataflow.dflow DEBUG: Task 313 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 313 submitted for App characterizeImage, waiting on task 312 parsl.dataflow.dflow DEBUG: Task 313 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 313 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 314 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 314 submitted for App calibrate, waiting on task 313 parsl.dataflow.dflow DEBUG: Task 314 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 314 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 315 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 315 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 315 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 315 has memoization hash c43f48ad74d4aecc40948ae61e18e28f parsl.dataflow.memoization INFO: Task 315 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f31d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 315 try 0 launched on executor multi with executor id 106 parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/0b8648c0-cdb8-4ef7-9da8-e344ca9724f1_isr_1125543_37.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/0b8648c0-cdb8-4ef7-9da8-e344ca9724f1_isr_1125543_37.stderr parsl.dataflow.dflow DEBUG: Task 316 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 316 submitted for App characterizeImage, waiting on task 315 parsl.dataflow.dflow DEBUG: Task 316 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 316 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 317 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 317 submitted for App calibrate, waiting on task 316 parsl.dataflow.dflow DEBUG: Task 317 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 317 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 318 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 318 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 318 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 318 has memoization hash e42316d03caaaa691283baeb57af35b9 parsl.dataflow.memoization INFO: Task 318 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f32020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 318 try 0 launched on executor multi with executor id 107 parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/56366522-7ce3-430e-9d6d-391766775429_isr_1125541_31.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/56366522-7ce3-430e-9d6d-391766775429_isr_1125541_31.stderr parsl.dataflow.dflow DEBUG: Task 319 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 319 submitted for App characterizeImage, waiting on task 318 parsl.dataflow.dflow DEBUG: Task 319 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 319 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 320 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 320 submitted for App calibrate, waiting on task 319 parsl.dataflow.dflow DEBUG: Task 320 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 320 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 321 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 321 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 321 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 321 has memoization hash 2a06f724f54c2600770aaf6712503515 parsl.dataflow.memoization INFO: Task 321 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f32340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 321 try 0 launched on executor multi with executor id 108 parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/d45aeff2-c5f3-4897-9fd3-dd5478dac7c9_isr_1125541_36.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/d45aeff2-c5f3-4897-9fd3-dd5478dac7c9_isr_1125541_36.stderr parsl.dataflow.dflow DEBUG: Task 322 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 322 submitted for App characterizeImage, waiting on task 321 parsl.dataflow.dflow DEBUG: Task 322 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 322 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 323 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 323 submitted for App calibrate, waiting on task 322 parsl.dataflow.dflow DEBUG: Task 323 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 323 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 324 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 324 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 324 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 324 has memoization hash a9899a7f95e68afbd7d21da433dbcd12 parsl.dataflow.memoization INFO: Task 324 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f32660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 324 try 0 launched on executor multi with executor id 109 parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/62171297-995f-4dfe-a47b-7d52417e64b9_isr_1125542_36.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/62171297-995f-4dfe-a47b-7d52417e64b9_isr_1125542_36.stderr parsl.dataflow.dflow DEBUG: Task 325 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 325 submitted for App characterizeImage, waiting on task 324 parsl.dataflow.dflow DEBUG: Task 325 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 325 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 326 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 326 submitted for App calibrate, waiting on task 325 parsl.dataflow.dflow DEBUG: Task 326 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 326 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 327 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 327 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 327 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 327 has memoization hash a8ada7cb7908e0a7533b396db5c6c778 parsl.dataflow.memoization INFO: Task 327 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f32980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 327 try 0 launched on executor multi with executor id 110 parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/21c7fc55-e256-46c0-912f-a8e4c16e96c1_isr_1125541_41.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/21c7fc55-e256-46c0-912f-a8e4c16e96c1_isr_1125541_41.stderr parsl.dataflow.dflow DEBUG: Task 328 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 328 submitted for App characterizeImage, waiting on task 327 parsl.dataflow.dflow DEBUG: Task 328 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 328 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 329 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 329 submitted for App calibrate, waiting on task 328 parsl.dataflow.dflow DEBUG: Task 329 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 329 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 330 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 330 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 330 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 330 has memoization hash e6768d8a281149117e3c836b828cbed5 parsl.dataflow.memoization INFO: Task 330 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f32ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 330 try 0 launched on executor multi with executor id 111 parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/9137e1f1-2b8a-467d-93a4-44f49fd640c3_isr_1125542_62.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/9137e1f1-2b8a-467d-93a4-44f49fd640c3_isr_1125542_62.stderr parsl.dataflow.dflow DEBUG: Task 331 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 331 submitted for App characterizeImage, waiting on task 330 parsl.dataflow.dflow DEBUG: Task 331 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 331 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 332 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 332 submitted for App calibrate, waiting on task 331 parsl.dataflow.dflow DEBUG: Task 332 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 332 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 333 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 333 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 333 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 333 has memoization hash 653a1793375140398670707e7c95a234 parsl.dataflow.memoization INFO: Task 333 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f32fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 333 try 0 launched on executor multi with executor id 112 parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/f10cf8dd-785e-4900-8728-2570425b6c97_isr_1125542_18.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/f10cf8dd-785e-4900-8728-2570425b6c97_isr_1125542_18.stderr parsl.dataflow.dflow DEBUG: Task 334 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 334 submitted for App characterizeImage, waiting on task 333 parsl.dataflow.dflow DEBUG: Task 334 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 334 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 335 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 335 submitted for App calibrate, waiting on task 334 parsl.dataflow.dflow DEBUG: Task 335 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 335 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 336 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 336 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 336 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 336 has memoization hash 5fd7664c112029aa0daf7786b74950d7 parsl.dataflow.memoization INFO: Task 336 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f332e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 336 try 0 launched on executor multi with executor id 113 parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/12b3fa96-2f81-45de-a6a6-609f0c4faddf_isr_1125543_3.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/12b3fa96-2f81-45de-a6a6-609f0c4faddf_isr_1125543_3.stderr parsl.dataflow.dflow DEBUG: Task 337 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 337 submitted for App characterizeImage, waiting on task 336 parsl.dataflow.dflow DEBUG: Task 337 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 337 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 338 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 338 submitted for App calibrate, waiting on task 337 parsl.dataflow.dflow DEBUG: Task 338 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 338 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 339 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 339 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 339 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 339 has memoization hash d19730d789c245f9b2118994ef0f4e76 parsl.dataflow.memoization INFO: Task 339 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f33600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 339 try 0 launched on executor multi with executor id 114 parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/c82525f8-b079-4f59-bc1a-2f41367611da_isr_1125543_28.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/c82525f8-b079-4f59-bc1a-2f41367611da_isr_1125543_28.stderr parsl.dataflow.dflow DEBUG: Task 340 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 340 submitted for App characterizeImage, waiting on task 339 parsl.dataflow.dflow DEBUG: Task 340 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 340 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 341 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 341 submitted for App calibrate, waiting on task 340 parsl.dataflow.dflow DEBUG: Task 341 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 341 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 342 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 342 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 342 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 342 has memoization hash 7f701bd84287da24567a320c6deca720 parsl.dataflow.memoization INFO: Task 342 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f33920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 342 try 0 launched on executor multi with executor id 115 parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/0626cc5a-73df-464a-9c9d-af99a6cae287_isr_1125541_19.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/0626cc5a-73df-464a-9c9d-af99a6cae287_isr_1125541_19.stderr parsl.dataflow.dflow DEBUG: Task 343 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 343 submitted for App characterizeImage, waiting on task 342 parsl.dataflow.dflow DEBUG: Task 343 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 343 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 344 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 344 submitted for App calibrate, waiting on task 343 parsl.dataflow.dflow DEBUG: Task 344 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 344 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 345 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 345 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 345 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 345 has memoization hash b31d2aa9a2b804da8496b2ac1bd3a83f parsl.dataflow.memoization INFO: Task 345 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f33c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 345 try 0 launched on executor multi with executor id 116 parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/c210bd7c-a407-4c16-96fc-7853a48fc3ff_isr_1125541_3.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/c210bd7c-a407-4c16-96fc-7853a48fc3ff_isr_1125541_3.stderr parsl.dataflow.dflow DEBUG: Task 346 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 346 submitted for App characterizeImage, waiting on task 345 parsl.dataflow.dflow DEBUG: Task 346 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 346 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 347 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 347 submitted for App calibrate, waiting on task 346 parsl.dataflow.dflow DEBUG: Task 347 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 347 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 348 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 348 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 348 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 348 has memoization hash 5493812eb6a6c68a1e07c77504359c44 parsl.dataflow.memoization INFO: Task 348 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f33f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 348 try 0 launched on executor multi with executor id 117 parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/ba2f6f06-929a-44cd-bc82-599c60852e4f_isr_1125542_37.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/ba2f6f06-929a-44cd-bc82-599c60852e4f_isr_1125542_37.stderr parsl.dataflow.dflow DEBUG: Task 349 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 349 submitted for App characterizeImage, waiting on task 348 parsl.dataflow.dflow DEBUG: Task 349 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 349 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 350 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 350 submitted for App calibrate, waiting on task 349 parsl.dataflow.dflow DEBUG: Task 350 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 350 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 351 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 351 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 351 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 351 has memoization hash 226de6ede28c9e0cb14af37d4f2b52d9 parsl.dataflow.memoization INFO: Task 351 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f782c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 351 try 0 launched on executor multi with executor id 118 parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/193b50d2-77f5-4ea8-ad6d-6f7fb23c67b3_isr_1125543_55.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/193b50d2-77f5-4ea8-ad6d-6f7fb23c67b3_isr_1125543_55.stderr parsl.dataflow.dflow DEBUG: Task 352 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 352 submitted for App characterizeImage, waiting on task 351 parsl.dataflow.dflow DEBUG: Task 352 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 352 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 353 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 353 submitted for App calibrate, waiting on task 352 parsl.dataflow.dflow DEBUG: Task 353 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 353 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 354 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 354 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 354 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 354 has memoization hash 94540944bfc0e38a342487f150fb5d3d parsl.dataflow.memoization INFO: Task 354 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f785e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 354 try 0 launched on executor multi with executor id 119 parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/68241d3d-257e-494c-bc5f-08580d2eba8c_isr_1125542_11.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/68241d3d-257e-494c-bc5f-08580d2eba8c_isr_1125542_11.stderr parsl.dataflow.dflow DEBUG: Task 355 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 355 submitted for App characterizeImage, waiting on task 354 parsl.dataflow.dflow DEBUG: Task 355 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 355 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 356 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 356 submitted for App calibrate, waiting on task 355 parsl.dataflow.dflow DEBUG: Task 356 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 356 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 357 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 357 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 357 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 357 has memoization hash 227251d1ca6253dbf3ae0ce45998adca parsl.dataflow.memoization INFO: Task 357 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f78900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 357 try 0 launched on executor multi with executor id 120 parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/617c1c4f-f657-4aed-a863-855cfec9a587_isr_1125542_47.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/617c1c4f-f657-4aed-a863-855cfec9a587_isr_1125542_47.stderr parsl.dataflow.dflow DEBUG: Task 358 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 358 submitted for App characterizeImage, waiting on task 357 parsl.dataflow.dflow DEBUG: Task 358 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 358 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 359 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 359 submitted for App calibrate, waiting on task 358 parsl.dataflow.dflow DEBUG: Task 359 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 359 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 360 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 360 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 360 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 360 has memoization hash cc894d26f4fff22ef97231b6e888ebeb parsl.dataflow.memoization INFO: Task 360 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f78c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 360 try 0 launched on executor multi with executor id 121 parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/e4a93610-5011-4537-ace1-bddf13e49b17_isr_1125543_10.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/e4a93610-5011-4537-ace1-bddf13e49b17_isr_1125543_10.stderr parsl.dataflow.dflow DEBUG: Task 361 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 361 submitted for App characterizeImage, waiting on task 360 parsl.dataflow.dflow DEBUG: Task 361 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 361 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 362 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 362 submitted for App calibrate, waiting on task 361 parsl.dataflow.dflow DEBUG: Task 362 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 362 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 363 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 363 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 363 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 363 has memoization hash de7242f41ce8aab377aa4c6717c6f645 parsl.dataflow.memoization INFO: Task 363 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f78f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 363 try 0 launched on executor multi with executor id 122 parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/b1951421-e7ab-402e-97c0-68ef5ccc237e_isr_1125541_57.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/b1951421-e7ab-402e-97c0-68ef5ccc237e_isr_1125541_57.stderr parsl.dataflow.dflow DEBUG: Task 364 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 364 submitted for App characterizeImage, waiting on task 363 parsl.dataflow.dflow DEBUG: Task 364 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 364 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 365 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 365 submitted for App calibrate, waiting on task 364 parsl.dataflow.dflow DEBUG: Task 365 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 365 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 366 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 366 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 366 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 366 has memoization hash cca4d2ded55e99199edc51f4adb08b03 parsl.dataflow.memoization INFO: Task 366 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f79260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 366 try 0 launched on executor multi with executor id 123 parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/9f5664d4-7109-480d-866e-6b9e1cb7a48e_isr_1125543_5.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/9f5664d4-7109-480d-866e-6b9e1cb7a48e_isr_1125543_5.stderr parsl.dataflow.dflow DEBUG: Task 367 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 367 submitted for App characterizeImage, waiting on task 366 parsl.dataflow.dflow DEBUG: Task 367 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 367 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 368 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 368 submitted for App calibrate, waiting on task 367 parsl.dataflow.dflow DEBUG: Task 368 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 368 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 369 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 369 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 369 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 369 has memoization hash f7119c25b9fa2a97c7e9b08fb4fcc503 parsl.dataflow.memoization INFO: Task 369 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f79580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 369 try 0 launched on executor multi with executor id 124 parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/69861417-d3c6-43d9-bbea-c896fdebb63a_isr_1125541_53.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/69861417-d3c6-43d9-bbea-c896fdebb63a_isr_1125541_53.stderr parsl.dataflow.dflow DEBUG: Task 370 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 370 submitted for App characterizeImage, waiting on task 369 parsl.dataflow.dflow DEBUG: Task 370 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 370 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 371 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 371 submitted for App calibrate, waiting on task 370 parsl.dataflow.dflow DEBUG: Task 371 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 371 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 372 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 372 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 372 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 372 has memoization hash 7613ea47e8a2ef0ec5b59cfa6529d42b parsl.dataflow.memoization INFO: Task 372 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f798a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 372 try 0 launched on executor multi with executor id 125 parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/37cc7f84-ed18-4103-9f7d-d9b3f9c51504_isr_1125542_39.stdout parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/37cc7f84-ed18-4103-9f7d-d9b3f9c51504_isr_1125542_39.stderr parsl.dataflow.dflow DEBUG: Task 373 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 373 submitted for App characterizeImage, waiting on task 372 parsl.dataflow.dflow DEBUG: Task 373 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 373 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 374 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 374 submitted for App calibrate, waiting on task 373 parsl.dataflow.dflow DEBUG: Task 374 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 374 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 375 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 375 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 375 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 375 has memoization hash c6ce1639b3532ac99d68ce0e56bbb736 parsl.dataflow.memoization INFO: Task 375 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f79bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 375 try 0 launched on executor multi with executor id 126 parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/3b58dfc5-c188-47fa-a1fa-8b53cd718d55_isr_1125541_22.stdout parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/3b58dfc5-c188-47fa-a1fa-8b53cd718d55_isr_1125541_22.stderr parsl.dataflow.dflow DEBUG: Task 376 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 376 submitted for App characterizeImage, waiting on task 375 parsl.dataflow.dflow DEBUG: Task 376 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 376 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 377 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 377 submitted for App calibrate, waiting on task 376 parsl.dataflow.dflow DEBUG: Task 377 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 377 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 378 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 378 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 378 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 378 has memoization hash f93085d0d9eb82163d3e73a2091cb56e parsl.dataflow.memoization INFO: Task 378 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f79ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 378 try 0 launched on executor multi with executor id 127 parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/4e95d249-6fe4-4f28-8409-8a48c8ff23e6_isr_1125543_30.stdout parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/4e95d249-6fe4-4f28-8409-8a48c8ff23e6_isr_1125543_30.stderr parsl.dataflow.dflow DEBUG: Task 379 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 379 submitted for App characterizeImage, waiting on task 378 parsl.dataflow.dflow DEBUG: Task 379 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 379 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 380 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 380 submitted for App calibrate, waiting on task 379 parsl.dataflow.dflow DEBUG: Task 380 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 380 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 381 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 381 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 381 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 381 has memoization hash 64d10e5a7cb1b9ea5fcf086a2b2832f7 parsl.dataflow.memoization INFO: Task 381 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f7a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 381 try 0 launched on executor multi with executor id 128 parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/2a9f0580-2675-4f0f-99ca-82b2a3786cda_isr_1125542_5.stdout parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/2a9f0580-2675-4f0f-99ca-82b2a3786cda_isr_1125542_5.stderr parsl.dataflow.dflow DEBUG: Task 382 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 382 submitted for App characterizeImage, waiting on task 381 parsl.dataflow.dflow DEBUG: Task 382 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 382 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 383 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 383 submitted for App calibrate, waiting on task 382 parsl.dataflow.dflow DEBUG: Task 383 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 383 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 384 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 384 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 384 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 384 has memoization hash d954fa987884b599b42b079ae38ff536 parsl.dataflow.memoization INFO: Task 384 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f7a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 384 try 0 launched on executor multi with executor id 129 parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/49e2b844-02cd-4232-9605-c82a179bb3c5_isr_1125542_57.stdout parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/49e2b844-02cd-4232-9605-c82a179bb3c5_isr_1125542_57.stderr parsl.dataflow.dflow DEBUG: Task 385 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 385 submitted for App characterizeImage, waiting on task 384 parsl.dataflow.dflow DEBUG: Task 385 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 385 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 386 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 386 submitted for App calibrate, waiting on task 385 parsl.dataflow.dflow DEBUG: Task 386 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 386 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 387 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 387 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 387 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 387 has memoization hash 2eb3cfca1c32e103a9e67dc2237bad1d parsl.dataflow.memoization INFO: Task 387 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f7a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 387 try 0 launched on executor multi with executor id 130 parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/10db20cf-0824-4eaa-96fa-5e83db321814_isr_1125541_47.stdout parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/10db20cf-0824-4eaa-96fa-5e83db321814_isr_1125541_47.stderr parsl.dataflow.dflow DEBUG: Task 388 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 388 submitted for App characterizeImage, waiting on task 387 parsl.dataflow.dflow DEBUG: Task 388 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 388 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 389 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 389 submitted for App calibrate, waiting on task 388 parsl.dataflow.dflow DEBUG: Task 389 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 389 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 390 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 390 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 390 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 390 has memoization hash 9aa6751a9144d34a7d29fdf71823840f parsl.dataflow.memoization INFO: Task 390 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f7ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 10:22:47 proc_lsst.multi:146 [INFO] found job 24403 in provider local proc_lsst.multi INFO: found job 24403 in provider local parsl.dataflow.dflow INFO: Parsl task 390 try 0 launched on executor multi with executor id 131 parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/9577b3f2-a81b-4aab-9e96-6afc5f8cdffa_isr_1125543_23.stdout parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/9577b3f2-a81b-4aab-9e96-6afc5f8cdffa_isr_1125543_23.stderr parsl.dataflow.dflow DEBUG: Task 391 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 391 submitted for App characterizeImage, waiting on task 390 parsl.dataflow.dflow DEBUG: Task 391 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 391 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 392 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 392 submitted for App calibrate, waiting on task 391 parsl.dataflow.dflow DEBUG: Task 392 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 392 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 393 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 393 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 393 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 393 has memoization hash 26b7a57150073ee92bf1f039c91f2bc1 parsl.dataflow.memoization INFO: Task 393 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbe200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 393 try 0 launched on executor multi with executor id 132 parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/df732600-76e5-4a01-a20f-870b46a54542_isr_1125543_29.stdout parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/df732600-76e5-4a01-a20f-870b46a54542_isr_1125543_29.stderr parsl.dataflow.dflow DEBUG: Task 394 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 394 submitted for App characterizeImage, waiting on task 393 parsl.dataflow.dflow DEBUG: Task 394 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 394 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 395 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 395 submitted for App calibrate, waiting on task 394 parsl.dataflow.dflow DEBUG: Task 395 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 395 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 396 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 396 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 396 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 396 has memoization hash d795c30d52e0bde95b22dd8ac3256180 parsl.dataflow.memoization INFO: Task 396 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbde40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 396 try 0 launched on executor multi with executor id 133 parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/7cfbfbcb-b50e-404e-8e18-744a981492f9_isr_1125543_33.stdout parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/7cfbfbcb-b50e-404e-8e18-744a981492f9_isr_1125543_33.stderr parsl.dataflow.dflow DEBUG: Task 397 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 397 submitted for App characterizeImage, waiting on task 396 parsl.dataflow.dflow DEBUG: Task 397 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 397 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 398 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 398 submitted for App calibrate, waiting on task 397 parsl.dataflow.dflow DEBUG: Task 398 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 398 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 399 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 399 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 399 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 399 has memoization hash 8ea6f5236b8fb880442d1f80ca2a5432 parsl.dataflow.memoization INFO: Task 399 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbdb20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 399 try 0 launched on executor multi with executor id 134 parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/88721219-f55a-4047-855e-14d58b59c332_isr_1125541_54.stdout parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/88721219-f55a-4047-855e-14d58b59c332_isr_1125541_54.stderr parsl.dataflow.dflow DEBUG: Task 400 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 400 submitted for App characterizeImage, waiting on task 399 parsl.dataflow.dflow DEBUG: Task 400 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 400 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 401 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 401 submitted for App calibrate, waiting on task 400 parsl.dataflow.dflow DEBUG: Task 401 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 401 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 402 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 402 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 402 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 402 has memoization hash 225014778466dc89fa465452734e4e54 parsl.dataflow.memoization INFO: Task 402 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbd800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 402 try 0 launched on executor multi with executor id 135 parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/4708e662-9fd3-4868-b489-a55dac48e23d_isr_1125542_59.stdout parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/4708e662-9fd3-4868-b489-a55dac48e23d_isr_1125542_59.stderr parsl.dataflow.dflow DEBUG: Task 403 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 403 submitted for App characterizeImage, waiting on task 402 parsl.dataflow.dflow DEBUG: Task 403 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 403 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 404 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 404 submitted for App calibrate, waiting on task 403 parsl.dataflow.dflow DEBUG: Task 404 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 404 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 405 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 405 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 405 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 405 has memoization hash b4f3a3560b8d7f969d757f67d7a96331 parsl.dataflow.memoization INFO: Task 405 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbd4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 405 try 0 launched on executor multi with executor id 136 parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/ee753b55-8e1f-4156-a346-3aabcb3b72ae_isr_1125541_16.stdout parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/ee753b55-8e1f-4156-a346-3aabcb3b72ae_isr_1125541_16.stderr parsl.dataflow.dflow DEBUG: Task 406 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 406 submitted for App characterizeImage, waiting on task 405 parsl.dataflow.dflow DEBUG: Task 406 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 406 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 407 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 407 submitted for App calibrate, waiting on task 406 parsl.dataflow.dflow DEBUG: Task 407 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 407 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 408 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 408 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 408 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 408 has memoization hash 5fa3bb81fc4e9d98a94e4308d2342efd parsl.dataflow.memoization INFO: Task 408 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbd1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 408 try 0 launched on executor multi with executor id 137 parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/61a61bbd-cd56-41f3-90ba-42969c0e3dee_isr_1125543_1.stdout parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/61a61bbd-cd56-41f3-90ba-42969c0e3dee_isr_1125543_1.stderr parsl.dataflow.dflow DEBUG: Task 409 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 409 submitted for App characterizeImage, waiting on task 408 parsl.dataflow.dflow DEBUG: Task 409 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 409 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 410 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 410 submitted for App calibrate, waiting on task 409 parsl.dataflow.dflow DEBUG: Task 410 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 410 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 411 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 411 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 411 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 411 has memoization hash 157ed4bb7ea40e87065c5da81020f457 parsl.dataflow.memoization INFO: Task 411 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 411 try 0 launched on executor multi with executor id 138 parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/ff368ccb-8905-40f7-a12d-78ae0eb6c5a6_isr_1125541_39.stdout parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/ff368ccb-8905-40f7-a12d-78ae0eb6c5a6_isr_1125541_39.stderr parsl.dataflow.dflow DEBUG: Task 412 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 412 submitted for App characterizeImage, waiting on task 411 parsl.dataflow.dflow DEBUG: Task 412 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 412 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 413 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 413 submitted for App calibrate, waiting on task 412 parsl.dataflow.dflow DEBUG: Task 413 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 413 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 414 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 414 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 414 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 414 has memoization hash 1108608fda7c1f6eb3fdf9b8cc184409 parsl.dataflow.memoization INFO: Task 414 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f7b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 414 try 0 launched on executor multi with executor id 139 parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/99994388-1301-450d-84d5-fd2b9b33ed12_isr_1125542_55.stdout parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/99994388-1301-450d-84d5-fd2b9b33ed12_isr_1125542_55.stderr parsl.dataflow.dflow DEBUG: Task 415 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 415 submitted for App characterizeImage, waiting on task 414 parsl.dataflow.dflow DEBUG: Task 415 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 415 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 416 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 416 submitted for App calibrate, waiting on task 415 parsl.dataflow.dflow DEBUG: Task 416 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 416 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 417 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 417 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 417 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 417 has memoization hash 65f6d8e23894512675d37cfd339163a5 parsl.dataflow.memoization INFO: Task 417 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f7b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 417 try 0 launched on executor multi with executor id 140 parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/cf8614b2-dda7-4ced-928b-f4da04138605_isr_1125542_3.stdout parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/cf8614b2-dda7-4ced-928b-f4da04138605_isr_1125542_3.stderr parsl.dataflow.dflow DEBUG: Task 418 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 418 submitted for App characterizeImage, waiting on task 417 parsl.dataflow.dflow DEBUG: Task 418 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 418 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 419 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 419 submitted for App calibrate, waiting on task 418 parsl.dataflow.dflow DEBUG: Task 419 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 419 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 420 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 420 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 420 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 420 has memoization hash 1219ec330e731ae3129123d8b97824d9 parsl.dataflow.memoization INFO: Task 420 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f7b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 420 try 0 launched on executor multi with executor id 141 parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/6d914876-b09f-4a88-92d5-dbc2b2f9615e_isr_1125542_8.stdout parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/6d914876-b09f-4a88-92d5-dbc2b2f9615e_isr_1125542_8.stderr parsl.dataflow.dflow DEBUG: Task 421 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 421 submitted for App characterizeImage, waiting on task 420 parsl.dataflow.dflow DEBUG: Task 421 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 421 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 422 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 422 submitted for App calibrate, waiting on task 421 parsl.dataflow.dflow DEBUG: Task 422 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 422 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 423 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 423 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 423 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 423 has memoization hash bdbdac7ff259b5f557c46105f4532494 parsl.dataflow.memoization INFO: Task 423 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f7bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 423 try 0 launched on executor multi with executor id 142 parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/4bf98154-e27e-42d9-b043-1ad71869def1_isr_1125543_46.stdout parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/4bf98154-e27e-42d9-b043-1ad71869def1_isr_1125543_46.stderr parsl.dataflow.dflow DEBUG: Task 424 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 424 submitted for App characterizeImage, waiting on task 423 parsl.dataflow.dflow DEBUG: Task 424 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 424 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 425 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 425 submitted for App calibrate, waiting on task 424 parsl.dataflow.dflow DEBUG: Task 425 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 425 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 426 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 426 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 426 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 426 has memoization hash d72e7d96d2d9d631eac74c7f93b826c7 parsl.dataflow.memoization INFO: Task 426 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbd120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 426 try 0 launched on executor multi with executor id 143 parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/2f5c430d-36a6-4612-8a80-61daa8692d32_isr_1125541_60.stdout parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/2f5c430d-36a6-4612-8a80-61daa8692d32_isr_1125541_60.stderr parsl.dataflow.dflow DEBUG: Task 427 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 427 submitted for App characterizeImage, waiting on task 426 parsl.dataflow.dflow DEBUG: Task 427 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 427 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 428 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 428 submitted for App calibrate, waiting on task 427 parsl.dataflow.dflow DEBUG: Task 428 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 428 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 429 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 429 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 429 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 429 has memoization hash 54e461f67dc3e9d62c0a66945ae98c50 parsl.dataflow.memoization INFO: Task 429 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbc040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 429 try 0 launched on executor multi with executor id 144 parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/f0ccde67-2183-4b9a-b6a0-c2f6297b68c6_isr_1125541_23.stdout parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/f0ccde67-2183-4b9a-b6a0-c2f6297b68c6_isr_1125541_23.stderr parsl.dataflow.dflow DEBUG: Task 430 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 430 submitted for App characterizeImage, waiting on task 429 parsl.dataflow.dflow DEBUG: Task 430 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 430 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 431 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 431 submitted for App calibrate, waiting on task 430 parsl.dataflow.dflow DEBUG: Task 431 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 431 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 432 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 432 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 432 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 432 has memoization hash 843175779a023c188b3bb572eabc9fab parsl.dataflow.memoization INFO: Task 432 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbc360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 432 try 0 launched on executor multi with executor id 145 parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/5bd43068-0d75-4ee6-b147-e42cfd55ceae_isr_1125542_45.stdout parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/5bd43068-0d75-4ee6-b147-e42cfd55ceae_isr_1125542_45.stderr parsl.dataflow.dflow DEBUG: Task 433 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 433 submitted for App characterizeImage, waiting on task 432 parsl.dataflow.dflow DEBUG: Task 433 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 433 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 434 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 434 submitted for App calibrate, waiting on task 433 parsl.dataflow.dflow DEBUG: Task 434 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 434 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 435 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 435 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 435 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 435 has memoization hash 83d5b0d537f819d9e5c7fbd19da78314 parsl.dataflow.memoization INFO: Task 435 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbc5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 435 try 0 launched on executor multi with executor id 146 parsl.dataflow.dflow INFO: Standard output for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/7eb5b213-8a74-4db7-97bc-eececbf20036_isr_1125543_45.stdout parsl.dataflow.dflow INFO: Standard error for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/7eb5b213-8a74-4db7-97bc-eececbf20036_isr_1125543_45.stderr parsl.dataflow.dflow DEBUG: Task 436 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 436 submitted for App characterizeImage, waiting on task 435 parsl.dataflow.dflow DEBUG: Task 436 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 436 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 437 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 437 submitted for App calibrate, waiting on task 436 parsl.dataflow.dflow DEBUG: Task 437 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 437 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 438 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 438 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 438 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 438 has memoization hash 4aefb6cbdbc99741072bbbf0e0da7712 parsl.dataflow.memoization INFO: Task 438 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbcae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 438 try 0 launched on executor multi with executor id 147 parsl.dataflow.dflow INFO: Standard output for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/7dc8c25e-2a0d-49f7-a3a6-902cfb874637_isr_1125543_61.stdout parsl.dataflow.dflow INFO: Standard error for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/7dc8c25e-2a0d-49f7-a3a6-902cfb874637_isr_1125543_61.stderr parsl.dataflow.dflow DEBUG: Task 439 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 439 submitted for App characterizeImage, waiting on task 438 parsl.dataflow.dflow DEBUG: Task 439 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 439 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 440 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 440 submitted for App calibrate, waiting on task 439 parsl.dataflow.dflow DEBUG: Task 440 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 440 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 441 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 441 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 441 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 441 has memoization hash 3b532fba7645212d132ed1dbb914330e parsl.dataflow.memoization INFO: Task 441 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbcd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 441 try 0 launched on executor multi with executor id 148 parsl.dataflow.dflow INFO: Standard output for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/3af5122a-51f2-4ae6-a6af-57ee8100eefd_isr_1125541_45.stdout parsl.dataflow.dflow INFO: Standard error for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/3af5122a-51f2-4ae6-a6af-57ee8100eefd_isr_1125541_45.stderr parsl.dataflow.dflow DEBUG: Task 442 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 442 submitted for App characterizeImage, waiting on task 441 parsl.dataflow.dflow DEBUG: Task 442 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 442 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 443 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 443 submitted for App calibrate, waiting on task 442 parsl.dataflow.dflow DEBUG: Task 443 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 443 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 444 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 444 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 444 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 444 has memoization hash d9b22a277fa1700db31d6bb3df4607fa parsl.dataflow.memoization INFO: Task 444 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbcea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 149 parsl.dataflow.strategy DEBUG: Executor multi has 149 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 37 more blocks parsl.executors.status_handling INFO: Scaling out by 37 blocks parsl.executors.status_handling INFO: Allocated block ID 1 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1 2024-03-07 10:22:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 2024-03-07 10:22:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:22:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:22:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:22:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:22:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-07 10:22:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 444 try 0 launched on executor multi with executor id 149 parsl.dataflow.dflow INFO: Standard output for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/19502779-309c-46fd-8c3f-01bb30eee660_isr_1125541_40.stdout parsl.dataflow.dflow INFO: Standard error for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/19502779-309c-46fd-8c3f-01bb30eee660_isr_1125541_40.stderr parsl.dataflow.dflow DEBUG: Task 445 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 445 submitted for App characterizeImage, waiting on task 444 parsl.dataflow.dflow DEBUG: Task 445 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 445 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 446 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 446 submitted for App calibrate, waiting on task 445 parsl.dataflow.dflow DEBUG: Task 446 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 446 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 447 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 447 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 447 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 447 has memoization hash bc9bd110c2ef6ec158ef60d154955597 parsl.dataflow.memoization INFO: Task 447 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbe8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 447 try 0 launched on executor multi with executor id 150 parsl.dataflow.dflow INFO: Standard output for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/8b7e9faf-c6b6-48f8-ac57-a8f0c1971e92_isr_1125542_24.stdout parsl.dataflow.dflow INFO: Standard error for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/8b7e9faf-c6b6-48f8-ac57-a8f0c1971e92_isr_1125542_24.stderr parsl.dataflow.dflow DEBUG: Task 448 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 448 submitted for App characterizeImage, waiting on task 447 parsl.dataflow.dflow DEBUG: Task 448 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 448 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 449 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 449 submitted for App calibrate, waiting on task 448 parsl.dataflow.dflow DEBUG: Task 449 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 449 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 450 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 450 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 450 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 450 has memoization hash 1ce39a0f1f948ce1719d45f768f8857f parsl.dataflow.memoization INFO: Task 450 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 450 try 0 launched on executor multi with executor id 151 parsl.dataflow.dflow INFO: Standard output for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/a0cb96bd-24ad-46aa-b8a4-6c9d7ade28ff_isr_1125541_15.stdout parsl.dataflow.dflow INFO: Standard error for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/a0cb96bd-24ad-46aa-b8a4-6c9d7ade28ff_isr_1125541_15.stderr parsl.dataflow.dflow DEBUG: Task 451 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 451 submitted for App characterizeImage, waiting on task 450 parsl.dataflow.dflow DEBUG: Task 451 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 451 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 452 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 452 submitted for App calibrate, waiting on task 451 parsl.dataflow.dflow DEBUG: Task 452 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 452 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 453 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 453 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 453 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 453 has memoization hash 275e6552a19002d377a590ee0cf57a8e parsl.dataflow.memoization INFO: Task 453 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 453 try 0 launched on executor multi with executor id 152 parsl.dataflow.dflow INFO: Standard output for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/afad57de-6dfd-414d-bb45-3e5ae3ee6be4_isr_1125541_18.stdout parsl.dataflow.dflow INFO: Standard error for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/afad57de-6dfd-414d-bb45-3e5ae3ee6be4_isr_1125541_18.stderr parsl.dataflow.dflow DEBUG: Task 454 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 454 submitted for App characterizeImage, waiting on task 453 parsl.dataflow.dflow DEBUG: Task 454 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 454 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 455 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 455 submitted for App calibrate, waiting on task 454 parsl.dataflow.dflow DEBUG: Task 455 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 455 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 456 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 456 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 456 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 456 has memoization hash 0f089afdcd152567796dfe91247d3a91 parsl.dataflow.memoization INFO: Task 456 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbf240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 456 try 0 launched on executor multi with executor id 153 parsl.dataflow.dflow INFO: Standard output for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/61320e9e-fd5b-483f-8149-4b882b666c71_isr_1125541_34.stdout parsl.dataflow.dflow INFO: Standard error for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/61320e9e-fd5b-483f-8149-4b882b666c71_isr_1125541_34.stderr parsl.dataflow.dflow DEBUG: Task 457 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 457 submitted for App characterizeImage, waiting on task 456 parsl.dataflow.dflow DEBUG: Task 457 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 457 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 458 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 458 submitted for App calibrate, waiting on task 457 parsl.dataflow.dflow DEBUG: Task 458 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 458 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 459 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 459 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 459 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 459 has memoization hash b78f1a5c5e126589bc89fb933dff6664 parsl.dataflow.memoization INFO: Task 459 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfe700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 459 try 0 launched on executor multi with executor id 154 parsl.dataflow.dflow INFO: Standard output for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/b9ff6dca-d4b8-405c-b32d-a7add4eb3373_isr_1125543_47.stdout parsl.dataflow.dflow INFO: Standard error for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/b9ff6dca-d4b8-405c-b32d-a7add4eb3373_isr_1125543_47.stderr parsl.dataflow.dflow DEBUG: Task 460 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 460 submitted for App characterizeImage, waiting on task 459 parsl.dataflow.dflow DEBUG: Task 460 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 460 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 461 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 461 submitted for App calibrate, waiting on task 460 parsl.dataflow.dflow DEBUG: Task 461 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 461 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 462 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 462 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 462 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 462 has memoization hash 75986978e99b40adcb355ed87de31a4b parsl.dataflow.memoization INFO: Task 462 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfe3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 462 try 0 launched on executor multi with executor id 155 parsl.dataflow.dflow INFO: Standard output for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/c24b14a3-9513-46f8-b68f-d0018b9c0693_isr_1125543_21.stdout parsl.dataflow.dflow INFO: Standard error for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/c24b14a3-9513-46f8-b68f-d0018b9c0693_isr_1125543_21.stderr parsl.dataflow.dflow DEBUG: Task 463 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 463 submitted for App characterizeImage, waiting on task 462 parsl.dataflow.dflow DEBUG: Task 463 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 463 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 464 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 464 submitted for App calibrate, waiting on task 463 parsl.dataflow.dflow DEBUG: Task 464 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 464 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 465 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 465 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 465 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 465 has memoization hash 9b75bb89956717cadc3f86f8ba600f35 parsl.dataflow.memoization INFO: Task 465 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfe0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 465 try 0 launched on executor multi with executor id 156 parsl.dataflow.dflow INFO: Standard output for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/7df55f4c-9afb-4bd9-9895-b3ec1ca2bfc2_isr_1125543_49.stdout parsl.dataflow.dflow INFO: Standard error for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/7df55f4c-9afb-4bd9-9895-b3ec1ca2bfc2_isr_1125543_49.stderr parsl.dataflow.dflow DEBUG: Task 466 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 466 submitted for App characterizeImage, waiting on task 465 parsl.dataflow.dflow DEBUG: Task 466 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 466 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 467 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 467 submitted for App calibrate, waiting on task 466 parsl.dataflow.dflow DEBUG: Task 467 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 467 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 468 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 468 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 468 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 468 has memoization hash 87fa9002967079a79f292dc7ff02fe1b parsl.dataflow.memoization INFO: Task 468 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbf9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 468 try 0 launched on executor multi with executor id 157 parsl.dataflow.dflow INFO: Standard output for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/68d45cfc-3e63-49fa-94d4-b12f3209eec9_isr_1125541_25.stdout parsl.dataflow.dflow INFO: Standard error for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/68d45cfc-3e63-49fa-94d4-b12f3209eec9_isr_1125541_25.stderr parsl.dataflow.dflow DEBUG: Task 469 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 469 submitted for App characterizeImage, waiting on task 468 parsl.dataflow.dflow DEBUG: Task 469 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 469 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 470 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 470 submitted for App calibrate, waiting on task 469 parsl.dataflow.dflow DEBUG: Task 470 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 470 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 471 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 471 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 471 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 471 has memoization hash d19c9267fe4d0ae70fd4c2797cdf9a13 parsl.dataflow.memoization INFO: Task 471 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbfd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 471 try 0 launched on executor multi with executor id 158 parsl.dataflow.dflow INFO: Standard output for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/3a0073c5-9595-4847-b49d-c7f2656f4471_isr_1125542_41.stdout parsl.dataflow.dflow INFO: Standard error for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/3a0073c5-9595-4847-b49d-c7f2656f4471_isr_1125542_41.stderr parsl.dataflow.dflow DEBUG: Task 472 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 472 submitted for App characterizeImage, waiting on task 471 parsl.dataflow.dflow DEBUG: Task 472 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 472 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 473 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 473 submitted for App calibrate, waiting on task 472 parsl.dataflow.dflow DEBUG: Task 473 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 473 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 474 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 474 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 474 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 474 has memoization hash bbbe9b617c9311689b3fd4f2d6ac80a9 parsl.dataflow.memoization INFO: Task 474 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfe480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 474 try 0 launched on executor multi with executor id 159 parsl.dataflow.dflow INFO: Standard output for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/7c8b3ef6-6543-4e1d-a3ab-da54cca8adb1_isr_1125543_60.stdout parsl.dataflow.dflow INFO: Standard error for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/7c8b3ef6-6543-4e1d-a3ab-da54cca8adb1_isr_1125543_60.stderr parsl.dataflow.dflow DEBUG: Task 475 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 475 submitted for App characterizeImage, waiting on task 474 parsl.dataflow.dflow DEBUG: Task 475 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 475 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 476 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 476 submitted for App calibrate, waiting on task 475 parsl.dataflow.dflow DEBUG: Task 476 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 476 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 477 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 477 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 477 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 477 has memoization hash ecb0b17bba91458dd2ea6f7947f03b0a parsl.dataflow.memoization INFO: Task 477 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfc0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 477 try 0 launched on executor multi with executor id 160 parsl.dataflow.dflow INFO: Standard output for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/1a346cc0-06b4-487b-87fd-cb5e8d3032db_isr_1125542_30.stdout parsl.dataflow.dflow INFO: Standard error for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/1a346cc0-06b4-487b-87fd-cb5e8d3032db_isr_1125542_30.stderr parsl.dataflow.dflow DEBUG: Task 478 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 478 submitted for App characterizeImage, waiting on task 477 parsl.dataflow.dflow DEBUG: Task 478 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 478 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 479 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 479 submitted for App calibrate, waiting on task 478 parsl.dataflow.dflow DEBUG: Task 479 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 479 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 480 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 480 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 480 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 480 has memoization hash 24bd14f767ac2ad56894336e6ec6dddf parsl.dataflow.memoization INFO: Task 480 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfc220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 480 try 0 launched on executor multi with executor id 161 parsl.dataflow.dflow INFO: Standard output for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/dbebfd88-a6fe-4ad7-a3d8-50cd03fcd7d0_isr_1125541_58.stdout parsl.dataflow.dflow INFO: Standard error for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/dbebfd88-a6fe-4ad7-a3d8-50cd03fcd7d0_isr_1125541_58.stderr parsl.dataflow.dflow DEBUG: Task 481 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 481 submitted for App characterizeImage, waiting on task 480 parsl.dataflow.dflow DEBUG: Task 481 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 481 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 482 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 482 submitted for App calibrate, waiting on task 481 parsl.dataflow.dflow DEBUG: Task 482 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 482 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 483 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 483 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 483 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 483 has memoization hash 13442686cdc5056e00311c50744183f2 parsl.dataflow.memoization INFO: Task 483 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfc680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 483 try 0 launched on executor multi with executor id 162 parsl.dataflow.dflow INFO: Standard output for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/6d170fe3-7f39-4b34-af48-211af76a2d14_isr_1125543_53.stdout parsl.dataflow.dflow INFO: Standard error for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/6d170fe3-7f39-4b34-af48-211af76a2d14_isr_1125543_53.stderr parsl.dataflow.dflow DEBUG: Task 484 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 484 submitted for App characterizeImage, waiting on task 483 parsl.dataflow.dflow DEBUG: Task 484 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 484 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 485 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 485 submitted for App calibrate, waiting on task 484 parsl.dataflow.dflow DEBUG: Task 485 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 485 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 486 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 486 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 486 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 486 has memoization hash a9c353431f9457a4eab7059c934026c1 parsl.dataflow.memoization INFO: Task 486 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfc9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 486 try 0 launched on executor multi with executor id 163 parsl.dataflow.dflow INFO: Standard output for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/4f1707df-9900-4f2e-adb8-d07b2aa0ed4f_isr_1125542_44.stdout parsl.dataflow.dflow INFO: Standard error for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/4f1707df-9900-4f2e-adb8-d07b2aa0ed4f_isr_1125542_44.stderr parsl.dataflow.dflow DEBUG: Task 487 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 487 submitted for App characterizeImage, waiting on task 486 parsl.dataflow.dflow DEBUG: Task 487 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 487 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 488 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 488 submitted for App calibrate, waiting on task 487 parsl.dataflow.dflow DEBUG: Task 488 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 488 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 489 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 489 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 489 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 489 has memoization hash 7187be8841c48db85d15bbbdbf70d932 parsl.dataflow.memoization INFO: Task 489 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfcc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 489 try 0 launched on executor multi with executor id 164 parsl.dataflow.dflow INFO: Standard output for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/7d060252-c16a-44d3-a893-2eb73c34682b_isr_1125542_14.stdout parsl.dataflow.dflow INFO: Standard error for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/7d060252-c16a-44d3-a893-2eb73c34682b_isr_1125542_14.stderr parsl.dataflow.dflow DEBUG: Task 490 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 490 submitted for App characterizeImage, waiting on task 489 parsl.dataflow.dflow DEBUG: Task 490 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 490 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 491 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 491 submitted for App calibrate, waiting on task 490 parsl.dataflow.dflow DEBUG: Task 491 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 491 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 492 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 492 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 492 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 492 has memoization hash 667efbd298b410847cb3d5be3c40572d parsl.dataflow.memoization INFO: Task 492 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfd120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 492 try 0 launched on executor multi with executor id 165 parsl.dataflow.dflow INFO: Standard output for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/6c9363e2-95de-449c-af4a-837cb36ab9e7_isr_1125543_40.stdout parsl.dataflow.dflow INFO: Standard error for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/6c9363e2-95de-449c-af4a-837cb36ab9e7_isr_1125543_40.stderr parsl.dataflow.dflow DEBUG: Task 493 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 493 submitted for App characterizeImage, waiting on task 492 parsl.dataflow.dflow DEBUG: Task 493 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 493 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 494 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 494 submitted for App calibrate, waiting on task 493 parsl.dataflow.dflow DEBUG: Task 494 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 494 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 495 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 495 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 495 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 495 has memoization hash ee424a2c83c11d9100f941aff216546e parsl.dataflow.memoization INFO: Task 495 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfd3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 495 try 0 launched on executor multi with executor id 166 parsl.dataflow.dflow INFO: Standard output for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/718af084-6fcc-44b3-990e-89169eec5142_isr_1125541_38.stdout parsl.dataflow.dflow INFO: Standard error for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/718af084-6fcc-44b3-990e-89169eec5142_isr_1125541_38.stderr parsl.dataflow.dflow DEBUG: Task 496 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 496 submitted for App characterizeImage, waiting on task 495 parsl.dataflow.dflow DEBUG: Task 496 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 496 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 497 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 497 submitted for App calibrate, waiting on task 496 parsl.dataflow.dflow DEBUG: Task 497 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 497 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 498 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 498 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 498 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 498 has memoization hash cc11ddb4c99ef3b9aa88104b6223c1d9 parsl.dataflow.memoization INFO: Task 498 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfdda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 498 try 0 launched on executor multi with executor id 167 parsl.dataflow.dflow INFO: Standard output for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/727915c6-5f7c-48da-9c28-9ede6575380e_isr_1125542_27.stdout parsl.dataflow.dflow INFO: Standard error for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/727915c6-5f7c-48da-9c28-9ede6575380e_isr_1125542_27.stderr parsl.dataflow.dflow DEBUG: Task 499 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 499 submitted for App characterizeImage, waiting on task 498 parsl.dataflow.dflow DEBUG: Task 499 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 499 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 500 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 500 submitted for App calibrate, waiting on task 499 parsl.dataflow.dflow DEBUG: Task 500 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 500 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 501 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 501 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 501 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 501 has memoization hash 122023fc7279e3c11d54a38e252d7af9 parsl.dataflow.memoization INFO: Task 501 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfda80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 501 try 0 launched on executor multi with executor id 168 parsl.dataflow.dflow INFO: Standard output for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/aded3a7c-9bdc-4ddd-9847-fd47ae2757ed_isr_1125541_8.stdout parsl.dataflow.dflow INFO: Standard error for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/aded3a7c-9bdc-4ddd-9847-fd47ae2757ed_isr_1125541_8.stderr parsl.dataflow.dflow DEBUG: Task 502 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 502 submitted for App characterizeImage, waiting on task 501 parsl.dataflow.dflow DEBUG: Task 502 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 502 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 503 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 503 submitted for App calibrate, waiting on task 502 parsl.dataflow.dflow DEBUG: Task 503 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 503 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 504 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 504 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 504 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 504 has memoization hash 843f3157545da7bac893b75742af4ddd parsl.dataflow.memoization INFO: Task 504 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfd760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 504 try 0 launched on executor multi with executor id 169 parsl.dataflow.dflow INFO: Standard output for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/3e2ef4f7-fa8a-4530-ac50-5af6e6cb9d7c_isr_1125542_26.stdout parsl.dataflow.dflow INFO: Standard error for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/3e2ef4f7-fa8a-4530-ac50-5af6e6cb9d7c_isr_1125542_26.stderr parsl.dataflow.dflow DEBUG: Task 505 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 505 submitted for App characterizeImage, waiting on task 504 parsl.dataflow.dflow DEBUG: Task 505 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 505 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 506 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 506 submitted for App calibrate, waiting on task 505 parsl.dataflow.dflow DEBUG: Task 506 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 506 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 507 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 507 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 507 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 507 has memoization hash 1dbecb4ff80396b16f40d6d036c71d6a parsl.dataflow.memoization INFO: Task 507 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfe980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 507 try 0 launched on executor multi with executor id 170 parsl.dataflow.dflow INFO: Standard output for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/b08ae198-cc4e-448e-a00a-ff91b339fa78_isr_1125542_46.stdout parsl.dataflow.dflow INFO: Standard error for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/b08ae198-cc4e-448e-a00a-ff91b339fa78_isr_1125542_46.stderr parsl.dataflow.dflow DEBUG: Task 508 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 508 submitted for App characterizeImage, waiting on task 507 parsl.dataflow.dflow DEBUG: Task 508 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 508 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 509 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 509 submitted for App calibrate, waiting on task 508 parsl.dataflow.dflow DEBUG: Task 509 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 509 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 510 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 510 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 510 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 510 has memoization hash eb9fa70a7e7340f8ef44796c4190e5a7 parsl.dataflow.memoization INFO: Task 510 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfeca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 510 try 0 launched on executor multi with executor id 171 parsl.dataflow.dflow INFO: Standard output for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/dfebb712-057c-4028-9de7-a7cb69bfadc0_isr_1125543_44.stdout parsl.dataflow.dflow INFO: Standard error for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/dfebb712-057c-4028-9de7-a7cb69bfadc0_isr_1125543_44.stderr parsl.dataflow.dflow DEBUG: Task 511 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 511 submitted for App characterizeImage, waiting on task 510 parsl.dataflow.dflow DEBUG: Task 511 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 511 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 512 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 512 submitted for App calibrate, waiting on task 511 parsl.dataflow.dflow DEBUG: Task 512 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 512 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 513 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 513 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 513 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 513 has memoization hash 42d72c81e261d9519883f07c34ab99bc parsl.dataflow.memoization INFO: Task 513 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfefc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 513 try 0 launched on executor multi with executor id 172 parsl.dataflow.dflow INFO: Standard output for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/d4b9a0aa-1b82-49b0-950e-87a0719a7186_isr_1125542_54.stdout parsl.dataflow.dflow INFO: Standard error for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/d4b9a0aa-1b82-49b0-950e-87a0719a7186_isr_1125542_54.stderr parsl.dataflow.dflow DEBUG: Task 514 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 514 submitted for App characterizeImage, waiting on task 513 parsl.dataflow.dflow DEBUG: Task 514 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 514 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 515 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 515 submitted for App calibrate, waiting on task 514 parsl.dataflow.dflow DEBUG: Task 515 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 515 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 516 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 516 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 516 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 516 has memoization hash 8876e772223f24172c087e6c1d708207 parsl.dataflow.memoization INFO: Task 516 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dff2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 516 try 0 launched on executor multi with executor id 173 parsl.dataflow.dflow INFO: Standard output for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/b714af61-e8b1-403c-8f44-b5f132718532_isr_1125541_13.stdout parsl.dataflow.dflow INFO: Standard error for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/b714af61-e8b1-403c-8f44-b5f132718532_isr_1125541_13.stderr parsl.dataflow.dflow DEBUG: Task 517 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 517 submitted for App characterizeImage, waiting on task 516 parsl.dataflow.dflow DEBUG: Task 517 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 517 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 518 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 518 submitted for App calibrate, waiting on task 517 parsl.dataflow.dflow DEBUG: Task 518 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 518 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 519 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 519 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 519 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 519 has memoization hash e0ba54dd01f671fa468ce085ce1bf1d5 parsl.dataflow.memoization INFO: Task 519 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dff600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 519 try 0 launched on executor multi with executor id 174 parsl.dataflow.dflow INFO: Standard output for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/f574e8a1-b4f4-4b31-8ef2-5ad1926ffd99_isr_1125542_21.stdout parsl.dataflow.dflow INFO: Standard error for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/f574e8a1-b4f4-4b31-8ef2-5ad1926ffd99_isr_1125542_21.stderr parsl.dataflow.dflow DEBUG: Task 520 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 520 submitted for App characterizeImage, waiting on task 519 parsl.dataflow.dflow DEBUG: Task 520 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 520 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 521 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 521 submitted for App calibrate, waiting on task 520 parsl.dataflow.dflow DEBUG: Task 521 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 521 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 522 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 522 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 522 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 522 has memoization hash 3bcca5f836a8253c0c1a7034aa28565b parsl.dataflow.memoization INFO: Task 522 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dff920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 522 try 0 launched on executor multi with executor id 175 parsl.dataflow.dflow INFO: Standard output for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/e18f27f8-3241-4be8-bf69-9c83faac0fe9_isr_1125541_52.stdout parsl.dataflow.dflow INFO: Standard error for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/e18f27f8-3241-4be8-bf69-9c83faac0fe9_isr_1125541_52.stderr parsl.dataflow.dflow DEBUG: Task 523 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 523 submitted for App characterizeImage, waiting on task 522 parsl.dataflow.dflow DEBUG: Task 523 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 523 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 524 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 524 submitted for App calibrate, waiting on task 523 parsl.dataflow.dflow DEBUG: Task 524 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 524 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 525 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 525 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 525 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 525 has memoization hash 7d91488ca9f40bfab26972ac44d684ac parsl.dataflow.memoization INFO: Task 525 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dffc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 525 try 0 launched on executor multi with executor id 176 parsl.dataflow.dflow INFO: Standard output for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/74f4f115-8b6e-4d9f-ae1a-d18d32c582e2_isr_1125541_5.stdout parsl.dataflow.dflow INFO: Standard error for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/74f4f115-8b6e-4d9f-ae1a-d18d32c582e2_isr_1125541_5.stderr parsl.dataflow.dflow DEBUG: Task 526 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 526 submitted for App characterizeImage, waiting on task 525 parsl.dataflow.dflow DEBUG: Task 526 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 526 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 527 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 527 submitted for App calibrate, waiting on task 526 parsl.dataflow.dflow DEBUG: Task 527 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 527 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 528 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 528 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 528 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 528 has memoization hash a64062dac82aba8b0fa5aa0269f33757 parsl.dataflow.memoization INFO: Task 528 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 528 try 0 launched on executor multi with executor id 177 parsl.dataflow.dflow INFO: Standard output for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/10ea726b-4e18-4b91-bb9f-1a4f3383fae5_isr_1125542_22.stdout parsl.dataflow.dflow INFO: Standard error for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/10ea726b-4e18-4b91-bb9f-1a4f3383fae5_isr_1125542_22.stderr parsl.dataflow.dflow DEBUG: Task 529 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 529 submitted for App characterizeImage, waiting on task 528 parsl.dataflow.dflow DEBUG: Task 529 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 529 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 530 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 530 submitted for App calibrate, waiting on task 529 parsl.dataflow.dflow DEBUG: Task 530 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 530 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 531 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 531 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 531 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 531 has memoization hash 365f22383fb935e3453a30cfb5bdea46 parsl.dataflow.memoization INFO: Task 531 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811e542c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 531 try 0 launched on executor multi with executor id 178 parsl.dataflow.dflow INFO: Standard output for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/c92d9f01-cc7c-4aa0-8fbe-af9f25fcc24b_isr_1125543_16.stdout parsl.dataflow.dflow INFO: Standard error for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/c92d9f01-cc7c-4aa0-8fbe-af9f25fcc24b_isr_1125543_16.stderr parsl.dataflow.dflow DEBUG: Task 532 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 532 submitted for App characterizeImage, waiting on task 531 parsl.dataflow.dflow DEBUG: Task 532 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 532 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 533 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 533 submitted for App calibrate, waiting on task 532 parsl.dataflow.dflow DEBUG: Task 533 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 533 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 534 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 534 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 534 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 534 has memoization hash 8f6b35b8769947f7da6368709b57f598 parsl.dataflow.memoization INFO: Task 534 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811e545e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 534 try 0 launched on executor multi with executor id 179 parsl.dataflow.dflow INFO: Standard output for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/bb14c084-19ed-42a1-bdef-e556f3b09a4f_isr_1125543_25.stdout parsl.dataflow.dflow INFO: Standard error for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/bb14c084-19ed-42a1-bdef-e556f3b09a4f_isr_1125543_25.stderr parsl.dataflow.dflow DEBUG: Task 535 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 535 submitted for App characterizeImage, waiting on task 534 parsl.dataflow.dflow DEBUG: Task 535 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 535 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 536 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 536 submitted for App calibrate, waiting on task 535 parsl.dataflow.dflow DEBUG: Task 536 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 536 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 537 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 537 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 537 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 537 has memoization hash 7eb9d402367a0b9db95c33e1919df3e6 parsl.dataflow.memoization INFO: Task 537 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811e54900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 537 try 0 launched on executor multi with executor id 180 parsl.dataflow.dflow INFO: Standard output for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/27fe204f-5cea-414a-a4df-c5c049a02d2d_isr_1125542_33.stdout parsl.dataflow.dflow INFO: Standard error for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/27fe204f-5cea-414a-a4df-c5c049a02d2d_isr_1125542_33.stderr parsl.dataflow.dflow DEBUG: Task 538 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 538 submitted for App characterizeImage, waiting on task 537 parsl.dataflow.dflow DEBUG: Task 538 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 538 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 539 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 539 submitted for App calibrate, waiting on task 538 parsl.dataflow.dflow DEBUG: Task 539 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 539 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 540 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 540 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 540 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 540 has memoization hash 7ef5a4d6159f6ee2940a99efbaae0780 parsl.dataflow.memoization INFO: Task 540 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811e54c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 10:22:47 proc_lsst.multi:170 [INFO] [multi] job_id 16889823 parsl.dataflow.dflow INFO: Parsl task 540 try 0 launched on executor multi with executor id 181 parsl.dataflow.dflow INFO: Standard output for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/dc7f9abe-6435-4546-beb2-def57fda6527_isr_1125542_56.stdout parsl.dataflow.dflow INFO: Standard error for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/dc7f9abe-6435-4546-beb2-def57fda6527_isr_1125542_56.stderr parsl.dataflow.dflow DEBUG: Task 541 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 541 submitted for App characterizeImage, waiting on task 540 parsl.dataflow.dflow DEBUG: Task 541 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 541 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 542 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 542 submitted for App calibrate, waiting on task 541 parsl.dataflow.dflow DEBUG: Task 542 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 542 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 543 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 543 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 543 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 543 has memoization hash 2345c4d670ae91a04fbd9cd99da2d2e6 parsl.dataflow.memoization INFO: Task 543 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811e54d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) proc_lsst.multi INFO: [multi] job_id 16889823 2024-03-07 10:22:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:22:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:47 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16889823 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16889823 parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 16889823 parsl.executors.status_handling INFO: Allocated block ID 2 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2 2024-03-07 10:22:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 2024-03-07 10:22:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:22:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:22:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:22:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:22:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:22:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 543 try 0 launched on executor multi with executor id 182 parsl.dataflow.dflow INFO: Standard output for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/b7887d93-cbd3-4481-b395-1b03449fd3c2_isr_1125541_50.stdout parsl.dataflow.dflow INFO: Standard error for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/b7887d93-cbd3-4481-b395-1b03449fd3c2_isr_1125541_50.stderr parsl.dataflow.dflow DEBUG: Task 544 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 544 submitted for App characterizeImage, waiting on task 543 parsl.dataflow.dflow DEBUG: Task 544 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 544 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 545 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 545 submitted for App calibrate, waiting on task 544 parsl.dataflow.dflow DEBUG: Task 545 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 545 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 546 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 546 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 546 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 546 has memoization hash 46fb10abed5c44b8dc31a1a8dfccae21 parsl.dataflow.memoization INFO: Task 546 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811e55080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 546 try 0 launched on executor multi with executor id 183 parsl.dataflow.dflow INFO: Standard output for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/792d84ae-9dc1-4a77-9f46-27ffe95723fb_isr_1125541_11.stdout parsl.dataflow.dflow INFO: Standard error for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/792d84ae-9dc1-4a77-9f46-27ffe95723fb_isr_1125541_11.stderr parsl.dataflow.dflow DEBUG: Task 547 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 547 submitted for App characterizeImage, waiting on task 546 parsl.dataflow.dflow DEBUG: Task 547 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 547 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 548 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 548 submitted for App calibrate, waiting on task 547 parsl.dataflow.dflow DEBUG: Task 548 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 548 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 549 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 549 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 549 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 549 has memoization hash 5eb0d6556931082fc9e2a57c991e8012 parsl.dataflow.memoization INFO: Task 549 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811e553a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 549 try 0 launched on executor multi with executor id 184 parsl.dataflow.dflow INFO: Standard output for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/b5d4d438-e9c4-4bdc-8cd8-8cdf13dda8da_isr_1125541_56.stdout parsl.dataflow.dflow INFO: Standard error for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/b5d4d438-e9c4-4bdc-8cd8-8cdf13dda8da_isr_1125541_56.stderr parsl.dataflow.dflow DEBUG: Task 550 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 550 submitted for App characterizeImage, waiting on task 549 parsl.dataflow.dflow DEBUG: Task 550 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 550 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 551 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 551 submitted for App calibrate, waiting on task 550 parsl.dataflow.dflow DEBUG: Task 551 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 551 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 552 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 552 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 552 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 552 has memoization hash cd855c701822a217a3a240619599f977 parsl.dataflow.memoization INFO: Task 552 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811e556c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 552 try 0 launched on executor multi with executor id 185 parsl.dataflow.dflow INFO: Standard output for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/6d75e0ef-0c9e-4be1-8ab6-94ccaf31b0ce_isr_1125541_7.stdout parsl.dataflow.dflow INFO: Standard error for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/6d75e0ef-0c9e-4be1-8ab6-94ccaf31b0ce_isr_1125541_7.stderr parsl.dataflow.dflow DEBUG: Task 553 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 553 submitted for App characterizeImage, waiting on task 552 parsl.dataflow.dflow DEBUG: Task 553 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 553 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 554 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 554 submitted for App calibrate, waiting on task 553 parsl.dataflow.dflow DEBUG: Task 554 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 554 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 555 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 555 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 555 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 555 has memoization hash c5fe2571449bdc0abd843d3e3d3f00b7 parsl.dataflow.memoization INFO: Task 555 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811e559e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 555 try 0 launched on executor multi with executor id 186 parsl.dataflow.dflow INFO: Standard output for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/e4d82c4f-3770-4a83-98e1-f7e65c4643a8_isr_1125543_52.stdout parsl.dataflow.dflow INFO: Standard error for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/e4d82c4f-3770-4a83-98e1-f7e65c4643a8_isr_1125543_52.stderr parsl.dataflow.dflow DEBUG: Task 556 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 556 submitted for App characterizeImage, waiting on task 555 parsl.dataflow.dflow DEBUG: Task 556 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 556 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 557 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 557 submitted for App calibrate, waiting on task 556 parsl.dataflow.dflow DEBUG: Task 557 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 557 has outstanding dependencies, so launch_if_ready skipping 2024-03-07 10:22:48 proc_lsst.multi:170 [INFO] [multi] job_id 16889824 proc_lsst.multi INFO: [multi] job_id 16889824 2024-03-07 10:22:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 10:22:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:48 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16889824 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16889824 parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 16889824 parsl.executors.status_handling INFO: Allocated block ID 3 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3 2024-03-07 10:22:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 2024-03-07 10:22:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:22:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:22:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:22:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:22:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 10:22:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 10:22:48 proc_lsst.multi:170 [INFO] [multi] job_id 16889825 proc_lsst.multi INFO: [multi] job_id 16889825 2024-03-07 10:22:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 10:22:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:48 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16889825 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16889825 parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 16889825 parsl.executors.status_handling INFO: Allocated block ID 4 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4 2024-03-07 10:22:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 2024-03-07 10:22:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:22:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:22:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:22:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:22:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 10:22:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 10:22:48 proc_lsst.multi:170 [INFO] [multi] job_id 16889826 proc_lsst.multi INFO: [multi] job_id 16889826 2024-03-07 10:22:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 10:22:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:48 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16889826 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16889826 parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 16889826 parsl.executors.status_handling INFO: Allocated block ID 5 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5 2024-03-07 10:22:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 2024-03-07 10:22:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:22:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:22:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:22:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:22:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 10:22:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 10:22:48 proc_lsst.multi:170 [INFO] [multi] job_id 16889827 proc_lsst.multi INFO: [multi] job_id 16889827 2024-03-07 10:22:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 10:22:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:48 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16889827 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16889827 parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 16889827 parsl.executors.status_handling INFO: Allocated block ID 6 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6 2024-03-07 10:22:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 2024-03-07 10:22:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:22:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:22:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:22:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:22:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 10:22:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 10:22:48 proc_lsst.multi:170 [INFO] [multi] job_id 16889828 proc_lsst.multi INFO: [multi] job_id 16889828 2024-03-07 10:22:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-07 10:22:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:48 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16889828 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16889828 parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 16889828 parsl.executors.status_handling INFO: Allocated block ID 7 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7 2024-03-07 10:22:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 2024-03-07 10:22:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:22:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:22:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:22:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:22:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-07 10:22:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 10:22:49 proc_lsst.multi:170 [INFO] [multi] job_id 16889829 proc_lsst.multi INFO: [multi] job_id 16889829 2024-03-07 10:22:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-07 10:22:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:49 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16889829 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16889829 parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 16889829 parsl.executors.status_handling INFO: Allocated block ID 8 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8 2024-03-07 10:22:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 2024-03-07 10:22:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:22:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:22:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:22:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:22:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-07 10:22:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 10:22:49 proc_lsst.multi:170 [INFO] [multi] job_id 16889830 proc_lsst.multi INFO: [multi] job_id 16889830 2024-03-07 10:22:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-07 10:22:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:49 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16889830 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16889830 parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 16889830 parsl.executors.status_handling INFO: Allocated block ID 9 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9 2024-03-07 10:22:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 2024-03-07 10:22:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:22:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:22:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:22:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:22:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-07 10:22:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 10:22:50 proc_lsst.multi:170 [INFO] [multi] job_id 16889831 proc_lsst.multi INFO: [multi] job_id 16889831 2024-03-07 10:22:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-07 10:22:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:50 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16889831 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16889831 parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 16889831 parsl.executors.status_handling INFO: Allocated block ID 10 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10 2024-03-07 10:22:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 2024-03-07 10:22:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:22:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:22:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:22:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:22:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-07 10:22:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 10:22:50 proc_lsst.multi:170 [INFO] [multi] job_id 16889832 proc_lsst.multi INFO: [multi] job_id 16889832 2024-03-07 10:22:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-07 10:22:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:50 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16889832 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16889832 parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 16889832 parsl.executors.status_handling INFO: Allocated block ID 11 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11 2024-03-07 10:22:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 2024-03-07 10:22:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:22:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:22:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:22:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:22:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-07 10:22:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 10:22:52 proc_lsst.multi:170 [INFO] [multi] job_id 16889833 proc_lsst.multi INFO: [multi] job_id 16889833 2024-03-07 10:22:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-07 10:22:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:52 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16889833 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16889833 parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 16889833 parsl.executors.status_handling INFO: Allocated block ID 12 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12 2024-03-07 10:22:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 2024-03-07 10:22:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:22:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:22:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:22:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:22:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-07 10:22:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 10:22:52 proc_lsst.multi:170 [INFO] [multi] job_id 16889834 proc_lsst.multi INFO: [multi] job_id 16889834 2024-03-07 10:22:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-07 10:22:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:52 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16889834 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16889834 parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 16889834 parsl.executors.status_handling INFO: Allocated block ID 13 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13 2024-03-07 10:22:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 2024-03-07 10:22:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:22:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:22:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:22:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:22:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-07 10:22:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 10:22:53 proc_lsst.multi:170 [INFO] [multi] job_id 16889835 proc_lsst.multi INFO: [multi] job_id 16889835 2024-03-07 10:22:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-07 10:22:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:53 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16889835 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16889835 parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 16889835 parsl.executors.status_handling INFO: Allocated block ID 14 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14 2024-03-07 10:22:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 2024-03-07 10:22:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:22:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:22:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:22:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:22:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-07 10:22:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 10:22:53 proc_lsst.multi:170 [INFO] [multi] job_id 16889836 proc_lsst.multi INFO: [multi] job_id 16889836 2024-03-07 10:22:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-07 10:22:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:53 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16889836 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16889836 parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 16889836 parsl.executors.status_handling INFO: Allocated block ID 15 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15 2024-03-07 10:22:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 2024-03-07 10:22:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:22:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:22:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:22:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:22:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-07 10:22:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 10:22:54 proc_lsst.multi:170 [INFO] [multi] job_id 16889837 proc_lsst.multi INFO: [multi] job_id 16889837 2024-03-07 10:22:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-07 10:22:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:54 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16889837 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16889837 parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 16889837 parsl.executors.status_handling INFO: Allocated block ID 16 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16 2024-03-07 10:22:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 2024-03-07 10:22:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:22:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:22:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:22:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:22:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-07 10:22:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 10:22:54 proc_lsst.multi:170 [INFO] [multi] job_id 16889838 proc_lsst.multi INFO: [multi] job_id 16889838 2024-03-07 10:22:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-07 10:22:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:54 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16889838 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16889838 parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 16889838 parsl.executors.status_handling INFO: Allocated block ID 17 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17 2024-03-07 10:22:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 2024-03-07 10:22:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:22:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:22:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:22:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:22:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-07 10:22:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 10:22:54 proc_lsst.multi:170 [INFO] [multi] job_id 16889839 proc_lsst.multi INFO: [multi] job_id 16889839 2024-03-07 10:22:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-07 10:22:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:54 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16889839 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16889839 parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 16889839 parsl.executors.status_handling INFO: Allocated block ID 18 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18 2024-03-07 10:22:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 2024-03-07 10:22:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:22:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:22:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:22:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:22:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-07 10:22:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 10:22:54 proc_lsst.multi:170 [INFO] [multi] job_id 16889840 proc_lsst.multi INFO: [multi] job_id 16889840 2024-03-07 10:22:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-07 10:22:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:54 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16889840 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16889840 parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 16889840 parsl.executors.status_handling INFO: Allocated block ID 19 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19 2024-03-07 10:22:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 2024-03-07 10:22:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:22:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:22:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:22:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:22:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-07 10:22:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 10:22:54 proc_lsst.multi:170 [INFO] [multi] job_id 16889841 proc_lsst.multi INFO: [multi] job_id 16889841 2024-03-07 10:22:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-07 10:22:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:54 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16889841 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16889841 parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 16889841 parsl.executors.status_handling INFO: Allocated block ID 20 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20 2024-03-07 10:22:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 2024-03-07 10:22:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:22:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:22:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:22:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:22:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-07 10:22:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 10:22:55 proc_lsst.multi:170 [INFO] [multi] job_id 16889842 proc_lsst.multi INFO: [multi] job_id 16889842 2024-03-07 10:22:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 10:22:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:55 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16889842 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16889842 parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 16889842 parsl.executors.status_handling INFO: Allocated block ID 21 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21 2024-03-07 10:22:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 2024-03-07 10:22:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:22:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:22:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:22:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:22:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 10:22:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 10:22:55 proc_lsst.multi:170 [INFO] [multi] job_id 16889843 proc_lsst.multi INFO: [multi] job_id 16889843 2024-03-07 10:22:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-07 10:22:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:55 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16889843 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16889843 parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 16889843 parsl.executors.status_handling INFO: Allocated block ID 22 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22 2024-03-07 10:22:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 2024-03-07 10:22:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:22:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:22:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:22:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:22:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-07 10:22:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 10:22:55 proc_lsst.multi:170 [INFO] [multi] job_id 16889844 proc_lsst.multi INFO: [multi] job_id 16889844 2024-03-07 10:22:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-07 10:22:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:55 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16889844 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16889844 parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 16889844 parsl.executors.status_handling INFO: Allocated block ID 23 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23 2024-03-07 10:22:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 2024-03-07 10:22:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:22:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:22:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:22:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:22:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-07 10:22:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 10:22:55 proc_lsst.multi:170 [INFO] [multi] job_id 16889845 proc_lsst.multi INFO: [multi] job_id 16889845 2024-03-07 10:22:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-07 10:22:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:55 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16889845 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16889845 parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 16889845 parsl.executors.status_handling INFO: Allocated block ID 24 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24 2024-03-07 10:22:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 2024-03-07 10:22:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:22:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:22:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:22:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:22:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-07 10:22:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 10:22:56 proc_lsst.multi:170 [INFO] [multi] job_id 16889846 proc_lsst.multi INFO: [multi] job_id 16889846 2024-03-07 10:22:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-07 10:22:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:56 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16889846 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16889846 parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 16889846 parsl.executors.status_handling INFO: Allocated block ID 25 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25 2024-03-07 10:22:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 2024-03-07 10:22:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:22:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:22:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:22:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:22:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-07 10:22:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 10:22:56 proc_lsst.multi:170 [INFO] [multi] job_id 16889847 proc_lsst.multi INFO: [multi] job_id 16889847 2024-03-07 10:22:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-07 10:22:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:56 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16889847 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16889847 parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 16889847 parsl.executors.status_handling INFO: Allocated block ID 26 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26 2024-03-07 10:22:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 2024-03-07 10:22:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:22:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:22:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:22:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:22:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-07 10:22:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 10:22:57 proc_lsst.multi:170 [INFO] [multi] job_id 16889848 proc_lsst.multi INFO: [multi] job_id 16889848 2024-03-07 10:22:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-07 10:22:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:57 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16889848 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16889848 parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 16889848 parsl.executors.status_handling INFO: Allocated block ID 27 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27 2024-03-07 10:22:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 2024-03-07 10:22:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:22:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:22:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:22:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:22:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-07 10:22:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:22:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5d4e3f3f8aceb2a776bfefa1a563f40 with result from task 3 parsl.dataflow.dflow DEBUG: Task 3 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash 98c8131b984ca90f4f08919d2b36d7e5 parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134913a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 187 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/a99fb97d-df94-40f1-9f84-d17465b8ff28_characterizeImage_1125543_19.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/a99fb97d-df94-40f1-9f84-d17465b8ff28_characterizeImage_1125543_19.stderr parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/805b5321-2360-462b-9ce0-27c562c7de95_isr_1125543_19.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/805b5321-2360-462b-9ce0-27c562c7de95_isr_1125543_19.stderr parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd24ff0df55efc830f746ca70a2aa27a with result from task 0 parsl.dataflow.dflow DEBUG: Task 0 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1 has memoization hash 32b468b441b7b25556e9c43977b6ae81 parsl.dataflow.memoization INFO: Task 1 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149813490ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 188 parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/9f1e39ae-2d1f-4269-87fa-1968fceea5b6_characterizeImage_1125543_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/9f1e39ae-2d1f-4269-87fa-1968fceea5b6_characterizeImage_1125543_62.stderr parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/4bf76bac-1005-4152-a7d7-fcae9edc155e_isr_1125543_62.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/4bf76bac-1005-4152-a7d7-fcae9edc155e_isr_1125543_62.stderr 2024-03-07 10:23:07 proc_lsst.multi:170 [INFO] [multi] job_id 16889852 proc_lsst.multi INFO: [multi] job_id 16889852 2024-03-07 10:23:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-07 10:23:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:23:07 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16889852 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16889852 parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 16889852 parsl.executors.status_handling INFO: Allocated block ID 28 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28 2024-03-07 10:23:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 2024-03-07 10:23:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:23:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:23:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:23:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-07 10:23:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:23:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 10:23:08 proc_lsst.multi:170 [INFO] [multi] job_id 16889853 proc_lsst.multi INFO: [multi] job_id 16889853 2024-03-07 10:23:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-07 10:23:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:23:08 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16889853 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16889853 parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 16889853 parsl.executors.status_handling INFO: Allocated block ID 29 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29 2024-03-07 10:23:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 2024-03-07 10:23:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:23:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:23:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:23:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-07 10:23:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:23:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 10:23:08 proc_lsst.multi:170 [INFO] [multi] job_id 16889854 proc_lsst.multi INFO: [multi] job_id 16889854 2024-03-07 10:23:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-07 10:23:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:23:08 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16889854 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16889854 parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 16889854 parsl.executors.status_handling INFO: Allocated block ID 30 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30 2024-03-07 10:23:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 2024-03-07 10:23:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:23:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:23:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:23:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-07 10:23:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:23:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 10:23:09 proc_lsst.multi:170 [INFO] [multi] job_id 16889855 proc_lsst.multi INFO: [multi] job_id 16889855 2024-03-07 10:23:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 10:23:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:23:09 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16889855 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16889855 parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 16889855 parsl.executors.status_handling INFO: Allocated block ID 31 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31 2024-03-07 10:23:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 2024-03-07 10:23:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:23:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:23:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:23:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 10:23:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:23:09 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-07 10:23:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 10:23:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 10:23:09 proc_lsst.multi:170 [INFO] [multi] job_id 16889856 proc_lsst.multi INFO: [multi] job_id 16889856 2024-03-07 10:23:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:23:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 10:23:09 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16889856 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16889856 parsl.executors.status_handling DEBUG: Launched block 31 on executor multi with job ID 16889856 parsl.executors.status_handling INFO: Allocated block ID 32 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32 2024-03-07 10:23:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 2024-03-07 10:23:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:23:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:23:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:23:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 10:23:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:23:09 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:23:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 10:23:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 420f47f290fbdaf2e8a39a5f6f3f6f60 with result from task 27 parsl.dataflow.dflow DEBUG: Task 27 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash 4ba87d1e2e9a26e3da9bbfc1f5d0ed84 parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149813493740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 189 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/1fa5e94f-b9b3-4e9f-8222-a315fa75f121_characterizeImage_1125541_2.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/1fa5e94f-b9b3-4e9f-8222-a315fa75f121_characterizeImage_1125541_2.stderr parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/fe61e302-b4c8-4664-ae50-5ccd6e0b34ee_isr_1125541_2.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/fe61e302-b4c8-4664-ae50-5ccd6e0b34ee_isr_1125541_2.stderr 2024-03-07 10:23:10 proc_lsst.multi:170 [INFO] [multi] job_id 16889857 proc_lsst.multi INFO: [multi] job_id 16889857 2024-03-07 10:23:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 10:23:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 10:23:10 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16889857 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16889857 parsl.executors.status_handling DEBUG: Launched block 32 on executor multi with job ID 16889857 parsl.executors.status_handling INFO: Allocated block ID 33 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33 2024-03-07 10:23:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 2024-03-07 10:23:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:23:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:23:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:23:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 10:23:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:23:10 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 10:23:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 10:23:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f297b13547e88aae194fc8199094ade2 with result from task 6 parsl.dataflow.dflow DEBUG: Task 6 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 7 has memoization hash cb37ab766fdf6e2272f149b7e85ec6c1 parsl.dataflow.memoization INFO: Task 7 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134918a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 190 parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/650854cd-95fa-4b60-9fa0-63e43a3d71c3_characterizeImage_1125542_38.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/650854cd-95fa-4b60-9fa0-63e43a3d71c3_characterizeImage_1125542_38.stderr parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/851df404-9543-44bc-bc67-0ad5a76a7a17_isr_1125542_38.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/851df404-9543-44bc-bc67-0ad5a76a7a17_isr_1125542_38.stderr 2024-03-07 10:23:11 proc_lsst.multi:170 [INFO] [multi] job_id 16889858 proc_lsst.multi INFO: [multi] job_id 16889858 2024-03-07 10:23:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 10:23:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 10:23:11 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16889858 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16889858 parsl.executors.status_handling DEBUG: Launched block 33 on executor multi with job ID 16889858 parsl.executors.status_handling INFO: Allocated block ID 34 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34 2024-03-07 10:23:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 2024-03-07 10:23:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:23:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:23:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:23:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 10:23:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:23:11 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 10:23:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 10:23:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68c4b37e9c9a63d5269263144a945b41 with result from task 9 parsl.dataflow.dflow DEBUG: Task 9 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash 7a7ae93bc72d2a6ee3e81f000095805f parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149813491bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 191 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/e8e108f9-68f6-4453-b369-9264be7a7909_characterizeImage_1125541_29.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/e8e108f9-68f6-4453-b369-9264be7a7909_characterizeImage_1125541_29.stderr parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/2318460f-9087-4c88-b4a3-161ab5f60bd1_isr_1125541_29.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/2318460f-9087-4c88-b4a3-161ab5f60bd1_isr_1125541_29.stderr 2024-03-07 10:23:12 proc_lsst.multi:170 [INFO] [multi] job_id 16889859 proc_lsst.multi INFO: [multi] job_id 16889859 2024-03-07 10:23:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 10:23:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 10:23:12 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16889859 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16889859 parsl.executors.status_handling DEBUG: Launched block 34 on executor multi with job ID 16889859 parsl.executors.status_handling INFO: Allocated block ID 35 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35 2024-03-07 10:23:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 2024-03-07 10:23:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:23:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:23:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:23:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 10:23:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:23:12 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 10:23:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 10:23:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 10:23:13 proc_lsst.multi:170 [INFO] [multi] job_id 16889860 proc_lsst.multi INFO: [multi] job_id 16889860 2024-03-07 10:23:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 10:23:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 10:23:13 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16889860 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16889860 parsl.executors.status_handling DEBUG: Launched block 35 on executor multi with job ID 16889860 parsl.executors.status_handling INFO: Allocated block ID 36 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36 2024-03-07 10:23:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 2024-03-07 10:23:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:23:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:23:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:23:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 10:23:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:23:13 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 10:23:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 10:23:13 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-07 10:23:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 10:23:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4a2b06e71e2e095ed74de4d588f4bc4 with result from task 24 parsl.dataflow.dflow DEBUG: Task 24 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 25 has memoization hash a1ae75f06d5fb540eb016e6ae42e1767 parsl.dataflow.memoization INFO: Task 25 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149813493a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 192 parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/6d8804b9-fd0d-453c-8cc2-8690c4925166_characterizeImage_1125541_26.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/6d8804b9-fd0d-453c-8cc2-8690c4925166_characterizeImage_1125541_26.stderr parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/c95b5212-b6b1-4eaa-bee2-a6160b9532c0_isr_1125541_26.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/c95b5212-b6b1-4eaa-bee2-a6160b9532c0_isr_1125541_26.stderr 2024-03-07 10:23:14 proc_lsst.multi:170 [INFO] [multi] job_id 16889861 proc_lsst.multi INFO: [multi] job_id 16889861 2024-03-07 10:23:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:23:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 10:23:14 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16889861 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16889861 parsl.executors.status_handling DEBUG: Launched block 36 on executor multi with job ID 16889861 parsl.executors.status_handling INFO: Allocated block ID 37 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37 2024-03-07 10:23:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 2024-03-07 10:23:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:23:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:23:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:23:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 10:23:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:23:14 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 10:23:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 10:23:14 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:23:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 10:23:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fec0bc1b672bd61b871e01de1a726613 with result from task 42 parsl.dataflow.dflow DEBUG: Task 42 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 43 has memoization hash 7090026edd87cf9b97a0e5d2646e7144 parsl.dataflow.memoization INFO: Task 43 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149813493ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 193 parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/e7e70b68-1337-4398-84f4-66273376df39_characterizeImage_1125543_8.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/e7e70b68-1337-4398-84f4-66273376df39_characterizeImage_1125543_8.stderr parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/66a2283c-1d20-4b76-96d1-f5c38149f4ba_isr_1125543_8.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/66a2283c-1d20-4b76-96d1-f5c38149f4ba_isr_1125543_8.stderr 2024-03-07 10:23:15 proc_lsst.multi:170 [INFO] [multi] job_id 16889862 proc_lsst.multi INFO: [multi] job_id 16889862 2024-03-07 10:23:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 10:23:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 10:23:15 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16889862 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16889862 parsl.executors.status_handling DEBUG: Launched block 37 on executor multi with job ID 16889862 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22643663051856 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 38, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 1/37 running/pending blocks, and 32 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 38 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 9 more blocks parsl.executors.status_handling INFO: Scaling out by 9 blocks parsl.executors.status_handling INFO: Allocated block ID 38 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-38 2024-03-07 10:23:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 2024-03-07 10:23:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:23:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:23:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:23:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 10:23:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:23:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 10:23:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 10:23:15 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 10:23:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 10:23:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 10:23:16 proc_lsst.multi:170 [INFO] [multi] job_id 16889863 proc_lsst.multi INFO: [multi] job_id 16889863 2024-03-07 10:23:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 10:23:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 10:23:16 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16889863 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16889863 parsl.executors.status_handling DEBUG: Launched block 38 on executor multi with job ID 16889863 parsl.executors.status_handling INFO: Allocated block ID 39 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-39 2024-03-07 10:23:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 2024-03-07 10:23:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:23:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:23:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:23:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 10:23:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:23:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 10:23:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 10:23:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 10:23:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 10:23:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c3e7c2e6b87ed31b3eeee700eb254dc with result from task 33 parsl.dataflow.dflow DEBUG: Task 33 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash 1bf438d44df45659d7376f68bd0b2067 parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149813493100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 194 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/19b54eb2-49be-480c-ab6e-a375d2ffbd56_characterizeImage_1125543_15.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/19b54eb2-49be-480c-ab6e-a375d2ffbd56_characterizeImage_1125543_15.stderr parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/720ea0d2-368f-4438-adbc-62614c90467f_isr_1125543_15.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/720ea0d2-368f-4438-adbc-62614c90467f_isr_1125543_15.stderr 2024-03-07 10:23:17 proc_lsst.multi:170 [INFO] [multi] job_id 16889864 proc_lsst.multi INFO: [multi] job_id 16889864 2024-03-07 10:23:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 10:23:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 10:23:17 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16889864 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16889864 parsl.executors.status_handling DEBUG: Launched block 39 on executor multi with job ID 16889864 parsl.executors.status_handling INFO: Allocated block ID 40 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-40 2024-03-07 10:23:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 2024-03-07 10:23:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:23:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:23:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:23:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 10:23:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:23:17 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 10:23:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 10:23:17 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 10:23:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 10:23:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b16c30ae4c93f4bbaea1865cde59f3af with result from task 30 parsl.dataflow.dflow DEBUG: Task 30 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 31 has memoization hash c931078d39c55770e995989b65c8f4d0 parsl.dataflow.memoization INFO: Task 31 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149813493420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 195 parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/5dc30535-2070-47fd-ad17-245b576c8b10_characterizeImage_1125542_50.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/5dc30535-2070-47fd-ad17-245b576c8b10_characterizeImage_1125542_50.stderr parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/aeb7b31f-c03d-4f4b-a5b6-100a8517add4_isr_1125542_50.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/aeb7b31f-c03d-4f4b-a5b6-100a8517add4_isr_1125542_50.stderr parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d452f89f8d73f2f3ee0fa236f624483 with result from task 36 parsl.dataflow.dflow DEBUG: Task 36 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 37 has memoization hash a9f127cd16c7a1e2480589857ca9576d parsl.dataflow.memoization INFO: Task 37 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149813492de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 196 parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/487dcdd1-07c4-4224-98f0-e15c06e63859_characterizeImage_1125543_50.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/487dcdd1-07c4-4224-98f0-e15c06e63859_characterizeImage_1125543_50.stderr parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/18eee0f8-3c10-4585-99fb-8290c0838c2e_isr_1125543_50.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/18eee0f8-3c10-4585-99fb-8290c0838c2e_isr_1125543_50.stderr parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90d0cb34c2c5be1ae0056d03e945c4c5 with result from task 60 parsl.dataflow.dflow DEBUG: Task 60 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 61 has memoization hash cc75b1776900b1fc32eab7eab92da95d parsl.dataflow.memoization INFO: Task 61 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b53a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 197 parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/cc1be6fe-47aa-44c6-a99c-202d1fa14d96_characterizeImage_1125541_27.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/cc1be6fe-47aa-44c6-a99c-202d1fa14d96_characterizeImage_1125541_27.stderr parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/2a6828e6-cd3a-4d51-93c0-bae7627e7e3b_isr_1125541_27.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/2a6828e6-cd3a-4d51-93c0-bae7627e7e3b_isr_1125541_27.stderr parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2c5e53b074e93574caf73a3cdd687eb with result from task 48 parsl.dataflow.dflow DEBUG: Task 48 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 49 has memoization hash 7409d0aa756f682a5edbf092cd1ab125 parsl.dataflow.memoization INFO: Task 49 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b4720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 198 parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/f97752e4-31ca-4cad-8584-6e83fb78ebc5_characterizeImage_1125543_26.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/f97752e4-31ca-4cad-8584-6e83fb78ebc5_characterizeImage_1125543_26.stderr parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/793bf2fe-d265-42cc-8e76-cffff8507a2a_isr_1125543_26.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/793bf2fe-d265-42cc-8e76-cffff8507a2a_isr_1125543_26.stderr parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a7f2a7ff0a66ce83a5964eed8e4c6cf with result from task 45 parsl.dataflow.dflow DEBUG: Task 45 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 46 has memoization hash ba9044c402c1fdf915e06a95ef66d4d6 parsl.dataflow.memoization INFO: Task 46 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b4400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 199 parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/245218ac-2d70-4462-b7ef-85977223dccf_characterizeImage_1125542_40.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/245218ac-2d70-4462-b7ef-85977223dccf_characterizeImage_1125542_40.stderr parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/71312d8a-ed98-4a88-9154-7dae2eecac39_isr_1125542_40.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/71312d8a-ed98-4a88-9154-7dae2eecac39_isr_1125542_40.stderr 2024-03-07 10:23:17 proc_lsst.multi:170 [INFO] [multi] job_id 16889865 proc_lsst.multi INFO: [multi] job_id 16889865 2024-03-07 10:23:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 10:23:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 10:23:17 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16889865 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16889865 parsl.executors.status_handling DEBUG: Launched block 40 on executor multi with job ID 16889865 parsl.executors.status_handling INFO: Allocated block ID 41 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-41 2024-03-07 10:23:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 2024-03-07 10:23:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:23:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:23:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:23:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 10:23:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:23:17 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 10:23:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 10:23:17 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 10:23:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 10:23:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c70c45f6ae22219c4ffa23239de091b with result from task 39 parsl.dataflow.dflow DEBUG: Task 39 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash bdb38e1d7c8156d8c146ac8711c03e14 parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149813492a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 200 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/ef7a984d-1716-47ce-8f2a-71c7ddf8f425_characterizeImage_1125541_59.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/ef7a984d-1716-47ce-8f2a-71c7ddf8f425_characterizeImage_1125541_59.stderr parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/04fadd06-5556-45db-b61e-c451c61ce1a5_isr_1125541_59.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/04fadd06-5556-45db-b61e-c451c61ce1a5_isr_1125541_59.stderr 2024-03-07 10:23:19 proc_lsst.multi:170 [INFO] [multi] job_id 16889866 proc_lsst.multi INFO: [multi] job_id 16889866 2024-03-07 10:23:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-07 10:23:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 10:23:19 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16889866 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16889866 parsl.executors.status_handling DEBUG: Launched block 41 on executor multi with job ID 16889866 parsl.executors.status_handling INFO: Allocated block ID 42 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-42 2024-03-07 10:23:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 2024-03-07 10:23:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:23:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:23:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:23:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 10:23:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:23:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 10:23:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 10:23:19 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-07 10:23:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 10:23:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 10:23:20 proc_lsst.multi:170 [INFO] [multi] job_id 16889867 proc_lsst.multi INFO: [multi] job_id 16889867 2024-03-07 10:23:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-07 10:23:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 10:23:20 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16889867 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16889867 parsl.executors.status_handling DEBUG: Launched block 42 on executor multi with job ID 16889867 parsl.executors.status_handling INFO: Allocated block ID 43 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-43 2024-03-07 10:23:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 2024-03-07 10:23:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:23:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:23:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:23:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 10:23:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:23:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 10:23:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 10:23:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-07 10:23:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 10:23:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 10:23:20 proc_lsst.multi:170 [INFO] [multi] job_id 16889868 proc_lsst.multi INFO: [multi] job_id 16889868 2024-03-07 10:23:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-07 10:23:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 10:23:20 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16889868 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16889868 parsl.executors.status_handling DEBUG: Launched block 43 on executor multi with job ID 16889868 parsl.executors.status_handling INFO: Allocated block ID 44 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-44 2024-03-07 10:23:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 2024-03-07 10:23:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:23:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:23:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:23:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 10:23:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:23:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 10:23:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 10:23:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-07 10:23:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 10:23:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6defd131ef8b524a75ff4e595fd606d with result from task 15 parsl.dataflow.dflow DEBUG: Task 15 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash f79ae5a46944ffa0795965554d0378c9 parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149813492340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 201 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/09517e27-ffc4-4af9-9b67-580e7029299a_characterizeImage_1125542_60.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/09517e27-ffc4-4af9-9b67-580e7029299a_characterizeImage_1125542_60.stderr parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/a960964f-61d7-46a4-94ca-acf35d2bb2e2_isr_1125542_60.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/a960964f-61d7-46a4-94ca-acf35d2bb2e2_isr_1125542_60.stderr parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79d417ecb2d231f2a4e5f56c25708069 with result from task 12 parsl.dataflow.dflow DEBUG: Task 12 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 13 has memoization hash 04cf994ee15a110c1e592fedf7a8841e parsl.dataflow.memoization INFO: Task 13 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149813491f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 202 parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/3cb571e7-d850-4e76-8385-50f414c88b80_characterizeImage_1125542_58.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/3cb571e7-d850-4e76-8385-50f414c88b80_characterizeImage_1125542_58.stderr parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/9ad16f3e-3405-4846-9779-aa429f5db5d0_isr_1125542_58.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/9ad16f3e-3405-4846-9779-aa429f5db5d0_isr_1125542_58.stderr parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9643916a9bfeae06d85f204837e9c343 with result from task 18 parsl.dataflow.dflow DEBUG: Task 18 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 19 has memoization hash d0f0172b734bba78dd9c3442d0f01213 parsl.dataflow.memoization INFO: Task 19 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149813492700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 203 parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/0b1aa223-bd70-44ee-97ea-bd51a0f7a410_characterizeImage_1125543_12.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/0b1aa223-bd70-44ee-97ea-bd51a0f7a410_characterizeImage_1125543_12.stderr parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/df0137b0-06e9-48a0-9ce9-3a82db8f6302_isr_1125543_12.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/df0137b0-06e9-48a0-9ce9-3a82db8f6302_isr_1125543_12.stderr 2024-03-07 10:23:21 proc_lsst.multi:170 [INFO] [multi] job_id 16889869 proc_lsst.multi INFO: [multi] job_id 16889869 2024-03-07 10:23:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-07 10:23:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 10:23:21 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16889869 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16889869 parsl.executors.status_handling DEBUG: Launched block 44 on executor multi with job ID 16889869 parsl.executors.status_handling INFO: Allocated block ID 45 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-45 2024-03-07 10:23:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 2024-03-07 10:23:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:23:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:23:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:23:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 10:23:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:23:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 10:23:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 10:23:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-07 10:23:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 10:23:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb7c6d93f9d19c091334eb8144448c9a with result from task 57 parsl.dataflow.dflow DEBUG: Task 57 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 58 has memoization hash 48eb64fce03e605769968dc6db80f1d3 parsl.dataflow.memoization INFO: Task 58 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b5080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 204 parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/1aa4c0b0-1830-44f8-ad6e-fa52db6e2705_characterizeImage_1125541_10.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/1aa4c0b0-1830-44f8-ad6e-fa52db6e2705_characterizeImage_1125541_10.stderr parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/39fe399f-2cc6-4269-92df-94dacd011566_isr_1125541_10.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/39fe399f-2cc6-4269-92df-94dacd011566_isr_1125541_10.stderr parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2fd7db9ae464f6735afc88eb8b4d776 with result from task 54 parsl.dataflow.dflow DEBUG: Task 54 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 55 has memoization hash 42320877e3b4cb007f5285f4e164ae5e parsl.dataflow.memoization INFO: Task 55 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b4d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 205 parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/ffd86f7f-6b4a-4f6c-8980-400bed9e8685_characterizeImage_1125541_44.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/ffd86f7f-6b4a-4f6c-8980-400bed9e8685_characterizeImage_1125541_44.stderr parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/c8ab690d-ed69-4612-83fb-1c8ad955eff9_isr_1125541_44.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/c8ab690d-ed69-4612-83fb-1c8ad955eff9_isr_1125541_44.stderr parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e6436c36d6e23be5bf3606e345cf427 with result from task 51 parsl.dataflow.dflow DEBUG: Task 51 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash 0ade0cc720fef4cd0cfda872368bd952 parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b4a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 206 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/b961916b-5350-414f-b0ce-fdd9e2576d31_characterizeImage_1125541_4.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/b961916b-5350-414f-b0ce-fdd9e2576d31_characterizeImage_1125541_4.stderr parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/c842de3a-cd6b-494e-9246-c31cb66bb608_isr_1125541_4.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/c842de3a-cd6b-494e-9246-c31cb66bb608_isr_1125541_4.stderr 2024-03-07 10:23:22 proc_lsst.multi:170 [INFO] [multi] job_id 16889870 proc_lsst.multi INFO: [multi] job_id 16889870 2024-03-07 10:23:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-07 10:23:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 10:23:22 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16889870 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16889870 parsl.executors.status_handling DEBUG: Launched block 45 on executor multi with job ID 16889870 parsl.executors.status_handling INFO: Allocated block ID 46 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-46 2024-03-07 10:23:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 2024-03-07 10:23:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 10:23:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 10:23:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 10:23:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 10:23:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 10:23:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 10:23:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 10:23:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 10:23:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-07 10:23:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 10:23:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a n3009,127.0.0.1,10.64.65.9,169.254.95.120,10.64.129.9,198.48.92.26 -p 0 -c 1.0 -m None --poll 10 --task_port=54775 --result_port=54360 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 10:23:22 proc_lsst.multi:170 [INFO] [multi] job_id 16889871 proc_lsst.multi INFO: [multi] job_id 16889871 2024-03-07 10:23:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-07 10:23:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 10:23:22 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16889871 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16889871 parsl.executors.status_handling DEBUG: Launched block 46 on executor multi with job ID 16889871 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22643663051856 2024-03-07 10:23:22 proc_lsst.multi:146 [INFO] found job 24403 in provider local proc_lsst.multi INFO: found job 24403 in provider local 2024-03-07 10:23:22 proc_lsst.multi:146 [INFO] found job 16889823 in provider astro proc_lsst.multi INFO: found job 16889823 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2cea4ee5f3ac67119cd796783c04cef9 with result from task 63 parsl.dataflow.dflow DEBUG: Task 63 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash 44403bd3b112d69fed86835d166c6dad parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b56c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 207 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/69a3c0c9-15b2-43f6-850e-1fb3967725fe_characterizeImage_1125543_38.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/69a3c0c9-15b2-43f6-850e-1fb3967725fe_characterizeImage_1125543_38.stderr parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/9f0d5ca1-6d0e-49a8-a2b5-d6aebc13ab97_isr_1125543_38.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/9f0d5ca1-6d0e-49a8-a2b5-d6aebc13ab97_isr_1125543_38.stderr parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34c07959d23fbdadc5fd6290f3309cbc with result from task 21 parsl.dataflow.dflow DEBUG: Task 21 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash 70ca8e34191ccafee926a0ba43c4500c parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149813493d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 208 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/2000f40d-8eda-44b7-bbea-08ecd9dbb933_characterizeImage_1125543_2.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/2000f40d-8eda-44b7-bbea-08ecd9dbb933_characterizeImage_1125543_2.stderr parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/e5faee53-0d84-4f16-993b-92d02b526756_isr_1125543_2.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/e5faee53-0d84-4f16-993b-92d02b526756_isr_1125543_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:23:25 proc_lsst.multi:146 [INFO] found job 16889824 in provider astro proc_lsst.multi INFO: found job 16889824 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d4f1e636f7c7987e8f75f7f11bfbc6a with result from task 84 parsl.dataflow.dflow DEBUG: Task 84 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 85 has memoization hash 27f5eb41313e35db05b6988ea66d3934 parsl.dataflow.memoization INFO: Task 85 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b6ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 209 parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/1fd8afc2-4144-4bc2-8187-66fb8cbacb69_characterizeImage_1125543_32.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/1fd8afc2-4144-4bc2-8187-66fb8cbacb69_characterizeImage_1125543_32.stderr parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/00f0be9b-32ce-470f-a0eb-2fe550fba8c1_isr_1125543_32.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/00f0be9b-32ce-470f-a0eb-2fe550fba8c1_isr_1125543_32.stderr parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ee7445941a24292cf7f6032765ce914 with result from task 99 parsl.dataflow.dflow DEBUG: Task 99 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash 8cccf12a7c4bc74c3bdbf121edb79ca7 parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b7c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 210 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/a332ba6f-3076-41fa-8c9f-ddab6f6ef9d6_characterizeImage_1125541_6.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/a332ba6f-3076-41fa-8c9f-ddab6f6ef9d6_characterizeImage_1125541_6.stderr parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/7d625fe9-1d77-43dc-87d5-bbbc66a9928b_isr_1125541_6.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/7d625fe9-1d77-43dc-87d5-bbbc66a9928b_isr_1125541_6.stderr parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd56d6002b812a6ef78eb653d5d732ba with result from task 96 parsl.dataflow.dflow DEBUG: Task 96 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 97 has memoization hash 438edc74a6cc3b90056f886c6a6a321c parsl.dataflow.memoization INFO: Task 97 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b7920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 211 parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/b0494e64-f1f7-4fd4-8c1f-8e87c1eb9fde_characterizeImage_1125543_39.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/b0494e64-f1f7-4fd4-8c1f-8e87c1eb9fde_characterizeImage_1125543_39.stderr parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/2cf93e14-fdf8-4bbc-90b3-ad3eaf024386_isr_1125543_39.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/2cf93e14-fdf8-4bbc-90b3-ad3eaf024386_isr_1125543_39.stderr parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8949447b080eaaff56134719c6a5134 with result from task 90 parsl.dataflow.dflow DEBUG: Task 90 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 91 has memoization hash 0314cc2a6fa773ab0f380f7fb6a27109 parsl.dataflow.memoization INFO: Task 91 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b72e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 212 parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/3f642c90-d4fb-4572-96d4-1f7e0d1da4ea_characterizeImage_1125541_28.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/3f642c90-d4fb-4572-96d4-1f7e0d1da4ea_characterizeImage_1125541_28.stderr parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/194df398-5cff-484f-be9c-cc1fe89b46e8_isr_1125541_28.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/194df398-5cff-484f-be9c-cc1fe89b46e8_isr_1125541_28.stderr parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5268857d1e52a169354e897e39a29228 with result from task 87 parsl.dataflow.dflow DEBUG: Task 87 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 88 has memoization hash 480108c646892d704a141e9b13b188ad parsl.dataflow.memoization INFO: Task 88 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b6fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 213 parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/61bb8955-9b9f-4f44-ad6b-048dd09597ac_characterizeImage_1125541_33.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/61bb8955-9b9f-4f44-ad6b-048dd09597ac_characterizeImage_1125541_33.stderr parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/86198a6c-8b7a-45ac-971e-3712e8fda3c8_isr_1125541_33.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/86198a6c-8b7a-45ac-971e-3712e8fda3c8_isr_1125541_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:23:26 proc_lsst.multi:146 [INFO] found job 16889825 in provider astro proc_lsst.multi INFO: found job 16889825 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8242c56d7220057fa1179cf94e558c4d with result from task 108 parsl.dataflow.dflow DEBUG: Task 108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 109 has memoization hash f04ca6497f4da78c1f7e5816ac661d69 parsl.dataflow.memoization INFO: Task 109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eac5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 214 parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/7b591715-5591-4cb2-ae0c-36726db40cec_characterizeImage_1125542_19.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/7b591715-5591-4cb2-ae0c-36726db40cec_characterizeImage_1125542_19.stderr parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/b4f10e75-540c-4814-8ad1-038a0a5e48e5_isr_1125542_19.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/b4f10e75-540c-4814-8ad1-038a0a5e48e5_isr_1125542_19.stderr parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e9925691120c8993d704505c5151d51 with result from task 105 parsl.dataflow.dflow DEBUG: Task 105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash ada5d702006821bb15d823ad89d539ba parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eac2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 215 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/0616b4dc-39f0-4043-88ca-b4579b300270_characterizeImage_1125541_62.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/0616b4dc-39f0-4043-88ca-b4579b300270_characterizeImage_1125541_62.stderr parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/fc8ee954-7a5e-4065-976c-233f536aae41_isr_1125541_62.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/fc8ee954-7a5e-4065-976c-233f536aae41_isr_1125541_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:23:28 proc_lsst.multi:146 [INFO] found job 16889826 in provider astro proc_lsst.multi INFO: found job 16889826 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc58928f786d36ac44bfacfe69e217f3 with result from task 111 parsl.dataflow.dflow DEBUG: Task 111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 112 has memoization hash 56eb355742d97ee42eaf7a6eff97a63a parsl.dataflow.memoization INFO: Task 112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eac900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 216 parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/3c6d60e9-2e23-4182-9a1f-b6c13b27bda3_characterizeImage_1125543_20.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/3c6d60e9-2e23-4182-9a1f-b6c13b27bda3_characterizeImage_1125543_20.stderr parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/1a435449-6d93-45fb-9003-c724f82559e7_isr_1125543_20.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/1a435449-6d93-45fb-9003-c724f82559e7_isr_1125543_20.stderr parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2132d2c39f98e8409dee94062c66f7d0 with result from task 93 parsl.dataflow.dflow DEBUG: Task 93 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 94 has memoization hash 3906a35294d93dc5e50a6d26ceac117f parsl.dataflow.memoization INFO: Task 94 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b7600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 217 parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/4c7882d9-cd30-48db-9424-6fc004e162ed_characterizeImage_1125541_37.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/4c7882d9-cd30-48db-9424-6fc004e162ed_characterizeImage_1125541_37.stderr parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/d2170479-9ad1-4e76-988b-3ed99dbe258b_isr_1125541_37.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/d2170479-9ad1-4e76-988b-3ed99dbe258b_isr_1125541_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:23:29 proc_lsst.multi:146 [INFO] found job 16889827 in provider astro proc_lsst.multi INFO: found job 16889827 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 908b111b0641fe5168726cc3726c04e5 with result from task 123 parsl.dataflow.dflow DEBUG: Task 123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 124 has memoization hash 59314a23c89e2216e78d8f3a4fe9b786 parsl.dataflow.memoization INFO: Task 124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ead580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 218 parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/044c47ec-8ed9-4411-bdda-2eaa74150658_characterizeImage_1125541_21.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/044c47ec-8ed9-4411-bdda-2eaa74150658_characterizeImage_1125541_21.stderr parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/23592d35-155b-4100-9d1d-aabd867391f4_isr_1125541_21.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/23592d35-155b-4100-9d1d-aabd867391f4_isr_1125541_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:23:30 proc_lsst.multi:146 [INFO] found job 16889828 in provider astro proc_lsst.multi INFO: found job 16889828 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40173fb64cf55cd6f93ccef7bc0b741c with result from task 114 parsl.dataflow.dflow DEBUG: Task 114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 115 has memoization hash 1aee49ed57402eaee37936fece0849f9 parsl.dataflow.memoization INFO: Task 115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eacc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 219 parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/5d442e07-1f34-4834-8284-b06d63b0e90e_characterizeImage_1125542_20.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/5d442e07-1f34-4834-8284-b06d63b0e90e_characterizeImage_1125542_20.stderr parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/748bf058-9379-48bc-8549-2cb31fedebe6_isr_1125542_20.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/748bf058-9379-48bc-8549-2cb31fedebe6_isr_1125542_20.stderr parsl.dataflow.dflow INFO: Task 147 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17e2cf50cb89b61b3d090487f459289a with result from task 147 parsl.dataflow.dflow DEBUG: Task 147 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 148 has memoization hash 259f9ef5b9e5f5b16e74f3bd6e9f300c parsl.dataflow.memoization INFO: Task 148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eaee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 220 parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/556c2692-5113-4bd6-9729-38b4a7b18426_characterizeImage_1125542_31.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/556c2692-5113-4bd6-9729-38b4a7b18426_characterizeImage_1125542_31.stderr parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/123f7c9c-ae73-4f00-a26b-d6970d633256_isr_1125542_31.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/123f7c9c-ae73-4f00-a26b-d6970d633256_isr_1125542_31.stderr parsl.dataflow.dflow INFO: Task 135 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a110cbe9496475cc87eab3d9b86f146c with result from task 135 parsl.dataflow.dflow DEBUG: Task 135 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 136 has memoization hash ac8316b1c68b496916db94e912384d9b parsl.dataflow.memoization INFO: Task 136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eae200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 221 parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/aeb187d5-9384-4f86-938a-ab685e1478d3_characterizeImage_1125543_24.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/aeb187d5-9384-4f86-938a-ab685e1478d3_characterizeImage_1125543_24.stderr parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/124b3a74-3f98-4016-abd8-f4db9a10797d_isr_1125543_24.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/124b3a74-3f98-4016-abd8-f4db9a10797d_isr_1125543_24.stderr parsl.dataflow.dflow INFO: Task 138 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9055ba5a4cad9910e2c72350a68f0046 with result from task 138 parsl.dataflow.dflow DEBUG: Task 138 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 139 has memoization hash 383be55477a7e0924ef404c8aa1a7e88 parsl.dataflow.memoization INFO: Task 139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eae520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 139 try 0 launched on executor multi with executor id 222 parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/fcbe2b7d-97c2-4b64-8f94-4a3a56772d80_characterizeImage_1125543_9.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/fcbe2b7d-97c2-4b64-8f94-4a3a56772d80_characterizeImage_1125543_9.stderr parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/ca8aa607-12f0-469b-8322-612680752d37_isr_1125543_9.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/ca8aa607-12f0-469b-8322-612680752d37_isr_1125543_9.stderr parsl.dataflow.dflow INFO: Task 153 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c22425bc04d62e66441c0bb5f5bd5c5f with result from task 153 parsl.dataflow.dflow DEBUG: Task 153 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 154 has memoization hash 008f66b8352491cf792d78488331b3f3 parsl.dataflow.memoization INFO: Task 154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eaf4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 223 parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/a80d86c0-f901-48d0-a2e5-6db335450326_characterizeImage_1125543_41.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/a80d86c0-f901-48d0-a2e5-6db335450326_characterizeImage_1125543_41.stderr parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/982ea051-4f70-4582-95e5-959594edae2e_isr_1125543_41.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/982ea051-4f70-4582-95e5-959594edae2e_isr_1125543_41.stderr parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 585ac54d2157788f9d4f9ec818edfc16 with result from task 102 parsl.dataflow.dflow DEBUG: Task 102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 103 has memoization hash 25202bcc451235f74ef5e0c00afbfd4a parsl.dataflow.memoization INFO: Task 103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b7f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 224 parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/dcc846d1-02f4-4aca-8c47-747b364632ab_characterizeImage_1125543_27.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/dcc846d1-02f4-4aca-8c47-747b364632ab_characterizeImage_1125543_27.stderr parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/1ceab569-76b6-4172-9787-949b7c73d5f3_isr_1125543_27.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/1ceab569-76b6-4172-9787-949b7c73d5f3_isr_1125543_27.stderr parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ebc1894eb45e5295bfec1f445c15e63b with result from task 78 parsl.dataflow.dflow DEBUG: Task 78 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 79 has memoization hash 4237b936c424b26f8e1d9ad3d8d20592 parsl.dataflow.memoization INFO: Task 79 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b6660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 225 parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/15a0a6de-2d5b-41be-ba2b-9a3874804a9f_characterizeImage_1125541_20.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/15a0a6de-2d5b-41be-ba2b-9a3874804a9f_characterizeImage_1125541_20.stderr parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/6a09bc2f-ee6b-48a1-ab05-e07448768c06_isr_1125541_20.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/6a09bc2f-ee6b-48a1-ab05-e07448768c06_isr_1125541_20.stderr parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fdab7ba8e09743eecf0a3e5a5dc57313 with result from task 81 parsl.dataflow.dflow DEBUG: Task 81 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash 9a06e2f9d4f6adcbf7a4b6ae9f2ed123 parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b6980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 226 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/e44a4715-c657-4955-8fd6-061b07829c0c_characterizeImage_1125542_34.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/e44a4715-c657-4955-8fd6-061b07829c0c_characterizeImage_1125542_34.stderr parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/94c38760-f56c-4087-88fd-08ee197ffcd4_isr_1125542_34.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/94c38760-f56c-4087-88fd-08ee197ffcd4_isr_1125542_34.stderr parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1cd77630fc2a9dbea05944e7a172ae55 with result from task 66 parsl.dataflow.dflow DEBUG: Task 66 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 67 has memoization hash 26d75d41dba3db9d93446fb1bff3929f parsl.dataflow.memoization INFO: Task 67 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b59e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 227 parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/24be0878-a9b3-4379-934e-9de37d70bf6d_characterizeImage_1125541_32.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/24be0878-a9b3-4379-934e-9de37d70bf6d_characterizeImage_1125541_32.stderr parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/fefae949-217e-4698-a2b3-85adf081ec0d_isr_1125541_32.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/fefae949-217e-4698-a2b3-85adf081ec0d_isr_1125541_32.stderr parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c61ccd1a78099b20d016a907ec894d8 with result from task 120 parsl.dataflow.dflow DEBUG: Task 120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 121 has memoization hash bb50db9f13537bf93733584b053700e1 parsl.dataflow.memoization INFO: Task 121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ead260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 228 parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/4e8791e5-4edd-4a9d-ac14-e1b97579fc05_characterizeImage_1125542_2.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/4e8791e5-4edd-4a9d-ac14-e1b97579fc05_characterizeImage_1125542_2.stderr parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/a18f6aeb-43b0-4eda-8e40-71ff6d1b1303_isr_1125542_2.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/a18f6aeb-43b0-4eda-8e40-71ff6d1b1303_isr_1125542_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:23:32 proc_lsst.multi:146 [INFO] found job 16889829 in provider astro proc_lsst.multi INFO: found job 16889829 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 156 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7fc0b841403d231f317405b431261f67 with result from task 156 parsl.dataflow.dflow DEBUG: Task 156 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 157 has memoization hash 749ab336978ea0ba7cb6d0932472405d parsl.dataflow.memoization INFO: Task 157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eaf7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 157 try 0 launched on executor multi with executor id 229 parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/1611386f-214e-4fef-b952-8af42d0f8f7e_characterizeImage_1125542_15.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/1611386f-214e-4fef-b952-8af42d0f8f7e_characterizeImage_1125542_15.stderr parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/ff807c24-b95b-4207-895b-116229c29a0e_isr_1125542_15.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/ff807c24-b95b-4207-895b-116229c29a0e_isr_1125542_15.stderr parsl.dataflow.dflow INFO: Task 141 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 506674e2f66933b63c8ceb9b2ae8d047 with result from task 141 parsl.dataflow.dflow DEBUG: Task 141 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 142 has memoization hash 8be8e96bfb3ba4f0633d609e3e2aba44 parsl.dataflow.memoization INFO: Task 142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eae840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 230 parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/25ba8335-b796-4c59-8f38-a05a0632cabd_characterizeImage_1125543_18.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/25ba8335-b796-4c59-8f38-a05a0632cabd_characterizeImage_1125543_18.stderr parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/17d19bf6-dfe4-40be-9520-35abe62c6c62_isr_1125543_18.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/17d19bf6-dfe4-40be-9520-35abe62c6c62_isr_1125543_18.stderr parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32bb69bedc6b42c627e501f2f8929f58 with result from task 69 parsl.dataflow.dflow DEBUG: Task 69 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash 10b6181370e03485340a4c0fc05af8b1 parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b5d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 231 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/6a8a9d36-e7fb-4f05-9d75-d051f4aa7324_characterizeImage_1125543_42.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/6a8a9d36-e7fb-4f05-9d75-d051f4aa7324_characterizeImage_1125543_42.stderr parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/fe0f060f-4c5f-4a2b-a366-f9c734197777_isr_1125543_42.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/fe0f060f-4c5f-4a2b-a366-f9c734197777_isr_1125543_42.stderr parsl.dataflow.dflow INFO: Task 144 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 050e2ee3143895db2b66bbba1c274622 with result from task 144 parsl.dataflow.dflow DEBUG: Task 144 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 145 has memoization hash d795b1889db9d7eeb8ee255b1c45c62b parsl.dataflow.memoization INFO: Task 145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eaeb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 145 try 0 launched on executor multi with executor id 232 parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/34e42774-6699-4a7c-a219-19e62bf778b2_characterizeImage_1125541_42.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/34e42774-6699-4a7c-a219-19e62bf778b2_characterizeImage_1125541_42.stderr parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/fc6fdf3e-d296-4734-8973-2587c8f3c87a_isr_1125541_42.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/fc6fdf3e-d296-4734-8973-2587c8f3c87a_isr_1125541_42.stderr parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cff6d20bbc7d2e96fbe4393a27ab17c3 with result from task 75 parsl.dataflow.dflow DEBUG: Task 75 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash d815d6cfcc611b37aa0f7f4afa1a39f9 parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b6340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 233 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/e35d2dbe-38b4-4ffd-bf7d-6a1ae16389f7_characterizeImage_1125543_35.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/e35d2dbe-38b4-4ffd-bf7d-6a1ae16389f7_characterizeImage_1125543_35.stderr parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/a3da51ab-b09f-4fd8-8f49-cd8f5d926a3e_isr_1125543_35.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/a3da51ab-b09f-4fd8-8f49-cd8f5d926a3e_isr_1125543_35.stderr parsl.dataflow.dflow INFO: Task 162 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49f59f5ae646842ba8ca216c679a96d8 with result from task 162 parsl.dataflow.dflow DEBUG: Task 162 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 163 has memoization hash 446c2059d7808563d9884f324f9b0477 parsl.dataflow.memoization INFO: Task 163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eafe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 163 try 0 launched on executor multi with executor id 234 parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/26224822-8c7b-4190-8099-92c0eed530ab_characterizeImage_1125542_35.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/26224822-8c7b-4190-8099-92c0eed530ab_characterizeImage_1125542_35.stderr parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/ed349b7f-4d20-4b6e-8209-fd770f7aaeed_isr_1125542_35.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/ed349b7f-4d20-4b6e-8209-fd770f7aaeed_isr_1125542_35.stderr parsl.dataflow.dflow INFO: Task 132 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30abbeedeccd2e6dd451353c07d5ae1f with result from task 132 parsl.dataflow.dflow DEBUG: Task 132 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 133 has memoization hash d86fbf323b9cbea6783ab70a593aca82 parsl.dataflow.memoization INFO: Task 133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eadee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 133 try 0 launched on executor multi with executor id 235 parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/a0c21ddd-0a50-45d2-a06a-eee8df892c9c_characterizeImage_1125543_13.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/a0c21ddd-0a50-45d2-a06a-eee8df892c9c_characterizeImage_1125543_13.stderr parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/dec91c94-6ab0-4f82-b99a-01a284b775c9_isr_1125543_13.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/dec91c94-6ab0-4f82-b99a-01a284b775c9_isr_1125543_13.stderr parsl.dataflow.dflow INFO: Task 168 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8e9d245e254198bf317e35cf4919c58 with result from task 168 parsl.dataflow.dflow DEBUG: Task 168 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 169 has memoization hash 87be9b9c2f404a076f0a3f1f02c50ba3 parsl.dataflow.memoization INFO: Task 169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed44a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 169 try 0 launched on executor multi with executor id 236 parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/36eca3ca-23f9-46e8-81e8-c55fefdd19e6_characterizeImage_1125541_51.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/36eca3ca-23f9-46e8-81e8-c55fefdd19e6_characterizeImage_1125541_51.stderr parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/53b08f40-6442-4d07-bbd9-a80bde0ec250_isr_1125541_51.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/53b08f40-6442-4d07-bbd9-a80bde0ec250_isr_1125541_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:23:33 proc_lsst.multi:146 [INFO] found job 16889830 in provider astro proc_lsst.multi INFO: found job 16889830 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 126 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 210b68d2e224af75cf0e9b760e3e12df with result from task 126 parsl.dataflow.dflow DEBUG: Task 126 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 127 has memoization hash d540db0ea0144ac993ececb67e9d6753 parsl.dataflow.memoization INFO: Task 127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ead8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 127 try 0 launched on executor multi with executor id 237 parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/2274241f-ebb6-4910-8b78-c93b80436f89_characterizeImage_1125542_43.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/2274241f-ebb6-4910-8b78-c93b80436f89_characterizeImage_1125542_43.stderr parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/920f8a24-ac55-4822-aa99-629f7a2270d1_isr_1125542_43.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/920f8a24-ac55-4822-aa99-629f7a2270d1_isr_1125542_43.stderr parsl.dataflow.dflow INFO: Task 129 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71cb68c73f22e247d7048c05c04eb464 with result from task 129 parsl.dataflow.dflow DEBUG: Task 129 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 130 has memoization hash 05a358fed292dfbc39ff02bf0936a6b4 parsl.dataflow.memoization INFO: Task 130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eadbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 238 parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/1874074c-b59b-4af4-8aa9-2c00fe9a218f_characterizeImage_1125542_4.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/1874074c-b59b-4af4-8aa9-2c00fe9a218f_characterizeImage_1125542_4.stderr parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/c1f5c588-c02f-473a-94bc-4390c02fa006_isr_1125542_4.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/c1f5c588-c02f-473a-94bc-4390c02fa006_isr_1125542_4.stderr parsl.dataflow.dflow INFO: Task 150 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2762660298cb9f5006a91010dbe2ed56 with result from task 150 parsl.dataflow.dflow DEBUG: Task 150 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 151 has memoization hash 65c99d32d9d599c39678eb42f2337216 parsl.dataflow.memoization INFO: Task 151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eaf1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 151 try 0 launched on executor multi with executor id 239 parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/6bde3c23-0c93-4e91-8f1a-10a4e90db362_characterizeImage_1125542_7.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/6bde3c23-0c93-4e91-8f1a-10a4e90db362_characterizeImage_1125542_7.stderr parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/033922a1-7260-4f58-ba52-ceb4ac82ef9c_isr_1125542_7.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/033922a1-7260-4f58-ba52-ceb4ac82ef9c_isr_1125542_7.stderr parsl.dataflow.dflow INFO: Task 165 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af301d8b1244fef82dea95746f9c257b with result from task 165 parsl.dataflow.dflow DEBUG: Task 165 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 166 has memoization hash 924bdc8dbe00c6e7d2298aa9b0d930ec parsl.dataflow.memoization INFO: Task 166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eaff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 240 parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/c211893b-c1c7-423f-87d2-62d3cdcfafe9_characterizeImage_1125543_11.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/c211893b-c1c7-423f-87d2-62d3cdcfafe9_characterizeImage_1125543_11.stderr parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/a61b60f1-0e7c-406d-9e55-ddd55578d708_isr_1125543_11.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/a61b60f1-0e7c-406d-9e55-ddd55578d708_isr_1125543_11.stderr parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c7716922ebb550ddcf53f65f4d9f55b with result from task 72 parsl.dataflow.dflow DEBUG: Task 72 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 73 has memoization hash ba940a1a5ae9ebc482b45df8c242ad59 parsl.dataflow.memoization INFO: Task 73 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b6020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 241 parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/c8980db3-776f-498e-aaf7-f58046145065_characterizeImage_1125541_12.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/c8980db3-776f-498e-aaf7-f58046145065_characterizeImage_1125541_12.stderr parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/c111edf7-bf1b-49c3-8055-eb45d3d4cacb_isr_1125541_12.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/c111edf7-bf1b-49c3-8055-eb45d3d4cacb_isr_1125541_12.stderr parsl.dataflow.dflow INFO: Task 159 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 104ed21ae6416a637b5bc87e53099949 with result from task 159 parsl.dataflow.dflow DEBUG: Task 159 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 160 has memoization hash f3357fb811a4759e1120787bd2446a7f parsl.dataflow.memoization INFO: Task 160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eafb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 242 parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/909a6832-175b-405e-81b6-4faa248b5a28_characterizeImage_1125542_61.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/909a6832-175b-405e-81b6-4faa248b5a28_characterizeImage_1125542_61.stderr parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/4529f0f8-22b8-4cb2-84a5-417ac8c724f0_isr_1125542_61.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/4529f0f8-22b8-4cb2-84a5-417ac8c724f0_isr_1125542_61.stderr parsl.dataflow.dflow INFO: Task 171 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f06b4f3dd1c945f08458b9bc46e9a493 with result from task 171 parsl.dataflow.dflow DEBUG: Task 171 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 172 has memoization hash 93c51288c830dff9ee1171a649ad493a parsl.dataflow.memoization INFO: Task 172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed47c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 243 parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/5735d9d8-af1e-4b4e-89fe-20a8b0b59b85_characterizeImage_1125541_35.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/5735d9d8-af1e-4b4e-89fe-20a8b0b59b85_characterizeImage_1125541_35.stderr parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/eb757be7-724d-425a-83e1-fff1ad34c7a1_isr_1125541_35.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/eb757be7-724d-425a-83e1-fff1ad34c7a1_isr_1125541_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:23:34 proc_lsst.multi:146 [INFO] found job 16889831 in provider astro proc_lsst.multi INFO: found job 16889831 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:23:35 proc_lsst.multi:146 [INFO] found job 16889832 in provider astro proc_lsst.multi INFO: found job 16889832 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:23:36 proc_lsst.multi:146 [INFO] found job 16889833 in provider astro proc_lsst.multi INFO: found job 16889833 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5291d3578a6965cf62f35478044cff0f with result from task 117 parsl.dataflow.dflow DEBUG: Task 117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash 186f55b18a0a62e4d55dd6398ab61a9c parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eacf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 244 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/2f582667-1dba-4753-882c-6641b0f6f8a8_characterizeImage_1125542_52.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/2f582667-1dba-4753-882c-6641b0f6f8a8_characterizeImage_1125542_52.stderr parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/d9d09485-d908-4a38-8605-86adcb02a7fe_isr_1125542_52.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/d9d09485-d908-4a38-8605-86adcb02a7fe_isr_1125542_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:23:37 proc_lsst.multi:146 [INFO] found job 16889834 in provider astro proc_lsst.multi INFO: found job 16889834 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 192 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f673d2c72298e4bef169dd644b035230 with result from task 192 parsl.dataflow.dflow DEBUG: Task 192 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 193 has memoization hash afd65580dc2c6b34c53b59dd9f2dc160 parsl.dataflow.memoization INFO: Task 193 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed5da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 193 try 0 launched on executor multi with executor id 245 parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/25d49a1c-4123-438f-ba15-c333cfa6face_characterizeImage_1125542_13.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/25d49a1c-4123-438f-ba15-c333cfa6face_characterizeImage_1125542_13.stderr parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/8a35b649-ce28-425f-98d6-7237bfd18fbd_isr_1125542_13.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/8a35b649-ce28-425f-98d6-7237bfd18fbd_isr_1125542_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:23:38 proc_lsst.multi:146 [INFO] found job 16889835 in provider astro proc_lsst.multi INFO: found job 16889835 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:23:38 proc_lsst.multi:146 [INFO] found job 16889836 in provider astro proc_lsst.multi INFO: found job 16889836 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 204 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0905b0904b20e27827b038d048ef001 with result from task 204 parsl.dataflow.dflow DEBUG: Task 204 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 205 has memoization hash f65a3cd949db2532bfbac6fc3dae2fdc parsl.dataflow.memoization INFO: Task 205 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed6a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 205 try 0 launched on executor multi with executor id 246 parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/759b503c-1c4f-4fa8-af74-51ee8573ed0e_characterizeImage_1125542_51.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/759b503c-1c4f-4fa8-af74-51ee8573ed0e_characterizeImage_1125542_51.stderr parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/2ad81a7b-e981-4d09-9181-ee2e21fc8abf_isr_1125542_51.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/2ad81a7b-e981-4d09-9181-ee2e21fc8abf_isr_1125542_51.stderr parsl.dataflow.dflow INFO: Task 174 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5260187c5f0c3a60d05e2432e9baac21 with result from task 174 parsl.dataflow.dflow DEBUG: Task 174 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 175 has memoization hash 365991100e5a48cef498eb3573764c15 parsl.dataflow.memoization INFO: Task 175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed4ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 175 try 0 launched on executor multi with executor id 247 parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/7d3f2651-e638-42e3-8150-39a6621e78ff_characterizeImage_1125543_6.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/7d3f2651-e638-42e3-8150-39a6621e78ff_characterizeImage_1125543_6.stderr parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/d2c6ea6d-9137-4787-9ecc-d6cd4f043643_isr_1125543_6.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/d2c6ea6d-9137-4787-9ecc-d6cd4f043643_isr_1125543_6.stderr parsl.dataflow.dflow INFO: Task 177 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66294f4cd5fa22b21c02fbf896337e28 with result from task 177 parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:23:39 proc_lsst.multi:146 [INFO] found job 16889837 in provider astro proc_lsst.multi INFO: found job 16889837 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow DEBUG: Task 177 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 178 has memoization hash 430ff1c8c5655c2f6b5c91c63fcbdb70 parsl.dataflow.memoization INFO: Task 178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed4e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 248 parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/72ddccf0-37eb-49fd-8fc8-c023534a94bb_characterizeImage_1125542_10.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/72ddccf0-37eb-49fd-8fc8-c023534a94bb_characterizeImage_1125542_10.stderr parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/6b1c30da-1bf8-4f33-bcda-5f08991500fd_isr_1125542_10.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/6b1c30da-1bf8-4f33-bcda-5f08991500fd_isr_1125542_10.stderr parsl.dataflow.dflow INFO: Task 180 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f90321872e46e43c54e0443d5252b65 with result from task 180 parsl.dataflow.dflow DEBUG: Task 180 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 181 has memoization hash e913f26274d93e4c16807668de2f8861 parsl.dataflow.memoization INFO: Task 181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed5120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 181 try 0 launched on executor multi with executor id 249 parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/e0f8c4d5-16b0-40c6-b955-271a8dc4ae7e_characterizeImage_1125542_42.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/e0f8c4d5-16b0-40c6-b955-271a8dc4ae7e_characterizeImage_1125542_42.stderr parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/43aee40d-5a1c-4747-ad4c-2cd9de70c068_isr_1125542_42.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/43aee40d-5a1c-4747-ad4c-2cd9de70c068_isr_1125542_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:23:40 proc_lsst.multi:146 [INFO] found job 16889838 in provider astro proc_lsst.multi INFO: found job 16889838 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 183 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0715d7a6d1edacca927a075f046d0675 with result from task 183 parsl.dataflow.dflow DEBUG: Task 183 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 184 has memoization hash eb5b1a5c88f948370d98b731bf77ed4d parsl.dataflow.memoization INFO: Task 184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed5440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 250 parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/ce482e12-4df9-4d97-a9f1-02f1098cbd52_characterizeImage_1125543_59.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/ce482e12-4df9-4d97-a9f1-02f1098cbd52_characterizeImage_1125543_59.stderr parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/2fdd7122-daf9-4988-9333-727c3c57166d_isr_1125543_59.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/2fdd7122-daf9-4988-9333-727c3c57166d_isr_1125543_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:23:40 proc_lsst.multi:146 [INFO] found job 16889839 in provider astro proc_lsst.multi INFO: found job 16889839 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:23:41 proc_lsst.multi:146 [INFO] found job 16889840 in provider astro proc_lsst.multi INFO: found job 16889840 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 210 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 232fb851d88ab91389f0e930ae992f76 with result from task 210 parsl.dataflow.dflow DEBUG: Task 210 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 211 has memoization hash 079c51588cd1a77f894ddd59eff087a6 parsl.dataflow.memoization INFO: Task 211 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed7060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 211 try 0 launched on executor multi with executor id 251 parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/cd673753-b560-41f0-81d8-b8ff70e6d79a_characterizeImage_1125542_29.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/cd673753-b560-41f0-81d8-b8ff70e6d79a_characterizeImage_1125542_29.stderr parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/1d49f597-ab8c-4bc2-8a5d-ce82072fe515_isr_1125542_29.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/1d49f597-ab8c-4bc2-8a5d-ce82072fe515_isr_1125542_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:23:42 proc_lsst.multi:146 [INFO] found job 16889841 in provider astro proc_lsst.multi INFO: found job 16889841 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 201 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87f82dfd235440dc3247d0be04a56ba3 with result from task 201 parsl.dataflow.dflow DEBUG: Task 201 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 202 has memoization hash 3a98fa7dc46b41b4697f47f0ca2f6b3c parsl.dataflow.memoization INFO: Task 202 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed6700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 202 try 0 launched on executor multi with executor id 252 parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/65a5853e-3bc2-4eac-843d-96d0f7414f4d_characterizeImage_1125543_56.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/65a5853e-3bc2-4eac-843d-96d0f7414f4d_characterizeImage_1125543_56.stderr parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/8547c195-a175-419e-8619-0df565950dfa_isr_1125543_56.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/8547c195-a175-419e-8619-0df565950dfa_isr_1125543_56.stderr parsl.dataflow.dflow INFO: Task 243 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1624f367bb419409697a2347d935dfc0 with result from task 243 parsl.dataflow.dflow DEBUG: Task 243 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 244 has memoization hash 011d9db9848a972549587695ae5cc352 parsl.dataflow.memoization INFO: Task 244 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efd300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 244 try 0 launched on executor multi with executor id 253 parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/9ddb77ec-8e17-4193-bd7a-98ff07d6b69e_characterizeImage_1125542_6.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/9ddb77ec-8e17-4193-bd7a-98ff07d6b69e_characterizeImage_1125542_6.stderr parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/eadfbc93-ac9b-4759-88a2-51176d7b8090_isr_1125542_6.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/eadfbc93-ac9b-4759-88a2-51176d7b8090_isr_1125542_6.stderr parsl.dataflow.dflow INFO: Task 198 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e459c19e4b29343208a88f1737f9738 with result from task 198 parsl.dataflow.dflow DEBUG: Task 198 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 199 has memoization hash 3bbff10f3552cc00cbe64ece77f7dcc8 parsl.dataflow.memoization INFO: Task 199 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed63e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 199 try 0 launched on executor multi with executor id 254 parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/81444af2-1fcc-4ec4-88f6-9c30ec0ba60b_characterizeImage_1125543_43.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/81444af2-1fcc-4ec4-88f6-9c30ec0ba60b_characterizeImage_1125543_43.stderr parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/f02bea9c-0862-441d-ac50-1577f83909b7_isr_1125543_43.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/f02bea9c-0862-441d-ac50-1577f83909b7_isr_1125543_43.stderr parsl.dataflow.dflow INFO: Task 252 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5f3878731509ee804468a371fa9ce5f with result from task 252 parsl.dataflow.dflow DEBUG: Task 252 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 253 has memoization hash 812b6eaf4f31e9123bdabfbdf75bbe38 parsl.dataflow.memoization INFO: Task 253 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efdc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 253 try 0 launched on executor multi with executor id 255 parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/8483b0c1-d27d-429b-9c20-c96aa1a457e6_characterizeImage_1125541_49.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/8483b0c1-d27d-429b-9c20-c96aa1a457e6_characterizeImage_1125541_49.stderr parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/f36a7d07-ae4c-4690-ab2f-955704e8cef7_isr_1125541_49.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/f36a7d07-ae4c-4690-ab2f-955704e8cef7_isr_1125541_49.stderr parsl.dataflow.dflow INFO: Task 246 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f4ade892f05c2efb1363e0c2c88bcae with result from task 246 parsl.dataflow.dflow DEBUG: Task 246 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 247 has memoization hash 2dcd7fa3e418cf90d493424b5de1af03 parsl.dataflow.memoization INFO: Task 247 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efd620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 247 try 0 launched on executor multi with executor id 256 parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/1689aa09-62b1-43bc-9ce1-e26c20d8ca73_characterizeImage_1125542_28.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/1689aa09-62b1-43bc-9ce1-e26c20d8ca73_characterizeImage_1125542_28.stderr parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/33fa5b81-ba68-4e2e-bd1e-5b575381b232_isr_1125542_28.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/33fa5b81-ba68-4e2e-bd1e-5b575381b232_isr_1125542_28.stderr parsl.dataflow.dflow INFO: Task 249 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1108b4bc71c452a28cf16288002252c with result from task 249 parsl.dataflow.dflow DEBUG: Task 249 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 250 has memoization hash 720056684c698991637df1ea74ba9ef6 parsl.dataflow.memoization INFO: Task 250 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efd940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 250 try 0 launched on executor multi with executor id 257 parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/f745f1ea-6776-4954-b769-4969bee4134c_characterizeImage_1125542_49.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/f745f1ea-6776-4954-b769-4969bee4134c_characterizeImage_1125542_49.stderr parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/9956a44d-5ba5-45fa-afbd-d60fc47bcbbc_isr_1125542_49.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/9956a44d-5ba5-45fa-afbd-d60fc47bcbbc_isr_1125542_49.stderr parsl.dataflow.dflow INFO: Task 261 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 373cc681d56afb76d78803e8ed0c69e2 with result from task 261 parsl.dataflow.dflow DEBUG: Task 261 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 262 has memoization hash c10d10b8fb2759ffa57ebb9beecccf98 parsl.dataflow.memoization INFO: Task 262 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efe5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 262 try 0 launched on executor multi with executor id 258 parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/6dd8c99c-afdd-495d-a62d-a8d1df8e0a12_characterizeImage_1125541_9.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/6dd8c99c-afdd-495d-a62d-a8d1df8e0a12_characterizeImage_1125541_9.stderr parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/89ae558c-7a3a-4f19-8ae5-310185ee02e4_isr_1125541_9.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/89ae558c-7a3a-4f19-8ae5-310185ee02e4_isr_1125541_9.stderr parsl.dataflow.dflow INFO: Task 255 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry feb248cbb06441ff3fea5261a68b7320 with result from task 255 parsl.dataflow.dflow DEBUG: Task 255 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 256 has memoization hash 8f57ce37445ce76d616822b391a444df parsl.dataflow.memoization INFO: Task 256 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efdf80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 256 try 0 launched on executor multi with executor id 259 parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/19d70ebb-e46e-4b12-9798-08bf16b4563b_characterizeImage_1125542_9.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/19d70ebb-e46e-4b12-9798-08bf16b4563b_characterizeImage_1125542_9.stderr parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/ed193144-db1d-4f6b-b064-e4216243c022_isr_1125542_9.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/ed193144-db1d-4f6b-b064-e4216243c022_isr_1125542_9.stderr parsl.dataflow.dflow INFO: Task 207 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07796be154ad47bfa0aced66509b1098 with result from task 207 parsl.dataflow.dflow DEBUG: Task 207 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 208 has memoization hash ada2b99af08dfeb37f0d465e28e405ca parsl.dataflow.memoization INFO: Task 208 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed6d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 208 try 0 launched on executor multi with executor id 260 parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/fa0d9485-6711-4c37-8f88-5fe8c7e46838_characterizeImage_1125541_48.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/fa0d9485-6711-4c37-8f88-5fe8c7e46838_characterizeImage_1125541_48.stderr parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/31bf069b-606a-44a6-a5f2-92b81bb4b5dc_isr_1125541_48.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/31bf069b-606a-44a6-a5f2-92b81bb4b5dc_isr_1125541_48.stderr parsl.dataflow.dflow INFO: Task 189 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 76cc6f323a78c2dbd1bc6e3b614a4a37 with result from task 189 parsl.dataflow.dflow DEBUG: Task 189 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 190 has memoization hash 3f9042a7e8c127a710b7b2260b6bc7bb parsl.dataflow.memoization INFO: Task 190 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed5a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 190 try 0 launched on executor multi with executor id 261 parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/cf7737a9-f3ca-4323-9e0c-a41aa91f09c2_characterizeImage_1125543_57.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/cf7737a9-f3ca-4323-9e0c-a41aa91f09c2_characterizeImage_1125543_57.stderr parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/4610b7ea-9c87-4a78-8ba2-0854f64d7f05_isr_1125543_57.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/4610b7ea-9c87-4a78-8ba2-0854f64d7f05_isr_1125543_57.stderr parsl.dataflow.dflow INFO: Task 216 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3cb09fba5a7a3fee5aa28132b168626f with result from task 216 parsl.dataflow.dflow DEBUG: Task 216 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 217 has memoization hash 1f5621cc4fd8a613e972afb438b3c0a0 parsl.dataflow.memoization INFO: Task 217 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed76a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 217 try 0 launched on executor multi with executor id 262 parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/0b574e50-53da-4078-a053-5f85c9e112ff_characterizeImage_1125542_32.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/0b574e50-53da-4078-a053-5f85c9e112ff_characterizeImage_1125542_32.stderr parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/7339672c-ae9e-4602-b7e3-319bafcd1b0a_isr_1125542_32.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/7339672c-ae9e-4602-b7e3-319bafcd1b0a_isr_1125542_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:23:43 proc_lsst.multi:146 [INFO] found job 16889842 in provider astro proc_lsst.multi INFO: found job 16889842 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 213 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43a03558eab2096d28b6acc0861dc080 with result from task 213 parsl.dataflow.dflow DEBUG: Task 213 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 214 has memoization hash b47b7007b1bccb5e97da513b9fe67e06 parsl.dataflow.memoization INFO: Task 214 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed7380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 214 try 0 launched on executor multi with executor id 263 parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/72f18c80-f999-4d22-8652-6112c3d01f0a_characterizeImage_1125542_48.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/72f18c80-f999-4d22-8652-6112c3d01f0a_characterizeImage_1125542_48.stderr parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/6319ecc3-222f-47de-8f29-c9fce168ec24_isr_1125542_48.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/6319ecc3-222f-47de-8f29-c9fce168ec24_isr_1125542_48.stderr parsl.dataflow.dflow INFO: Task 270 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7243399de7dec313e3140db43e96676 with result from task 270 parsl.dataflow.dflow DEBUG: Task 270 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 271 has memoization hash 9b4fa9ba9b3e9b206e9e92f3f92d8aa3 parsl.dataflow.memoization INFO: Task 271 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 271 try 0 launched on executor multi with executor id 264 parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/328aa6fc-2464-450b-84f8-019707c925e4_characterizeImage_1125543_36.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/328aa6fc-2464-450b-84f8-019707c925e4_characterizeImage_1125543_36.stderr parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/56b79f35-328d-4e06-9bbd-b180653f4bd0_isr_1125543_36.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/56b79f35-328d-4e06-9bbd-b180653f4bd0_isr_1125543_36.stderr parsl.dataflow.dflow INFO: Task 285 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e48e8be06cd0b01747b9c397541e1064 with result from task 285 parsl.dataflow.dflow DEBUG: Task 285 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 286 has memoization hash 2f89c092345a985ded52a980d6bf341f parsl.dataflow.memoization INFO: Task 286 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811effec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 286 try 0 launched on executor multi with executor id 265 parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/8f52e894-c89f-4ade-a261-ecaa4a38e820_characterizeImage_1125543_17.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/8f52e894-c89f-4ade-a261-ecaa4a38e820_characterizeImage_1125543_17.stderr parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/e55779fa-4853-4a1b-a83d-21bc25aa7343_isr_1125543_17.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/e55779fa-4853-4a1b-a83d-21bc25aa7343_isr_1125543_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:23:44 proc_lsst.multi:146 [INFO] found job 16889843 in provider astro proc_lsst.multi INFO: found job 16889843 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 195 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99268a62769be421707305bf42826054 with result from task 195 parsl.dataflow.dflow DEBUG: Task 195 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 196 has memoization hash 171f9864b96e34e0839c38f493bf4134 parsl.dataflow.memoization INFO: Task 196 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed60c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 196 try 0 launched on executor multi with executor id 266 parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/6bbb65e9-5d7f-4c11-993c-c917c9c63af3_characterizeImage_1125541_61.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/6bbb65e9-5d7f-4c11-993c-c917c9c63af3_characterizeImage_1125541_61.stderr parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/cea4f72d-3bd0-4851-854c-7153bfa3781e_isr_1125541_61.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/cea4f72d-3bd0-4851-854c-7153bfa3781e_isr_1125541_61.stderr parsl.dataflow.dflow INFO: Task 303 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ba8907b3828bbb741f75eef48853614 with result from task 303 parsl.dataflow.dflow DEBUG: Task 303 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 304 has memoization hash cd1c321c5b72eeb7fb5bdf0cc93e5959 parsl.dataflow.memoization INFO: Task 304 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f311c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 304 try 0 launched on executor multi with executor id 267 parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/8ad48c45-e8b4-46d5-ad90-820e3701b332_characterizeImage_1125541_46.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/8ad48c45-e8b4-46d5-ad90-820e3701b332_characterizeImage_1125541_46.stderr parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/2088ae61-58e3-4593-b349-f43b34bdffef_isr_1125541_46.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/2088ae61-58e3-4593-b349-f43b34bdffef_isr_1125541_46.stderr parsl.dataflow.dflow INFO: Task 186 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 88105c22d5608a1b56bdf66e038f2cc3 with result from task 186 parsl.dataflow.dflow DEBUG: Task 186 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 187 has memoization hash 91ec1e87085da9718657eee1cb447bff parsl.dataflow.memoization INFO: Task 187 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed5760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 187 try 0 launched on executor multi with executor id 268 parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/6aa20301-9608-4428-b4af-880920b11115_characterizeImage_1125541_43.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/6aa20301-9608-4428-b4af-880920b11115_characterizeImage_1125541_43.stderr parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/51b1c61b-2b93-4c3b-b26b-3af364d820de_isr_1125541_43.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/51b1c61b-2b93-4c3b-b26b-3af364d820de_isr_1125541_43.stderr parsl.dataflow.dflow INFO: Task 312 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1303aa9ee350822a5c3eaa72f5a7ee6 with result from task 312 parsl.dataflow.dflow DEBUG: Task 312 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 313 has memoization hash dfecc735cb016ba7834a04a025b55913 parsl.dataflow.memoization INFO: Task 313 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f31b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 313 try 0 launched on executor multi with executor id 269 parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/194fe1cf-aa35-45c1-88cf-f9622887344b_characterizeImage_1125541_17.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/194fe1cf-aa35-45c1-88cf-f9622887344b_characterizeImage_1125541_17.stderr parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/6fce3ea1-456e-459d-ac3f-345d10e7cf9c_isr_1125541_17.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/6fce3ea1-456e-459d-ac3f-345d10e7cf9c_isr_1125541_17.stderr parsl.dataflow.dflow INFO: Task 240 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 861654228e9bdd58c7f6ee7a629654d7 with result from task 240 parsl.dataflow.dflow DEBUG: Task 240 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 241 has memoization hash 4fc6208e805007c07f059b4888f05121 parsl.dataflow.memoization INFO: Task 241 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efcfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.dataflow.dflow INFO: Parsl task 241 try 0 launched on executor multi with executor id 270 parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/5a60539a-df56-4e9c-9d92-01e10386fe02_characterizeImage_1125541_30.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/5a60539a-df56-4e9c-9d92-01e10386fe02_characterizeImage_1125541_30.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/2b3ebb3c-9295-4f1b-a2c3-5ce98a6b1aae_isr_1125541_30.stdout parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/2b3ebb3c-9295-4f1b-a2c3-5ce98a6b1aae_isr_1125541_30.stderr parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:23:45 proc_lsst.multi:146 [INFO] found job 16889844 in provider astro proc_lsst.multi INFO: found job 16889844 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 309 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4171bf776044ffb651461f11319f6725 with result from task 309 parsl.dataflow.dflow DEBUG: Task 309 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 310 has memoization hash 929b477ca7acc2368346b7499ff27a4a parsl.dataflow.memoization INFO: Task 310 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f31800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 310 try 0 launched on executor multi with executor id 271 parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/1691dedf-2669-4324-baf6-78ba005df4e7_characterizeImage_1125542_16.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/1691dedf-2669-4324-baf6-78ba005df4e7_characterizeImage_1125542_16.stderr parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/fc720e8e-dc6f-40cf-ade5-91d655a1a5e1_isr_1125542_16.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/fc720e8e-dc6f-40cf-ade5-91d655a1a5e1_isr_1125542_16.stderr parsl.dataflow.dflow INFO: Task 291 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 659cc1457560720d246976b12608e171 with result from task 291 parsl.dataflow.dflow DEBUG: Task 291 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 292 has memoization hash c9eed7683fd4318a9675253666c4e290 parsl.dataflow.memoization INFO: Task 292 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f30540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 292 try 0 launched on executor multi with executor id 272 parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/ff527bed-cfc7-41f4-b1c4-35443522f817_characterizeImage_1125541_55.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/ff527bed-cfc7-41f4-b1c4-35443522f817_characterizeImage_1125541_55.stderr parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/803c99b0-0605-4b2e-9cec-033a144132ed_isr_1125541_55.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/803c99b0-0605-4b2e-9cec-033a144132ed_isr_1125541_55.stderr parsl.dataflow.dflow INFO: Task 273 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dace0a55e7fea54e1621583f9848cbf6 with result from task 273 parsl.dataflow.dflow DEBUG: Task 273 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 274 has memoization hash 14bdd080c84f72d7ac1643075aa547e3 parsl.dataflow.memoization INFO: Task 274 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eff240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 274 try 0 launched on executor multi with executor id 273 parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/5ff71989-ed8c-437b-bc33-5da799703f53_characterizeImage_1125541_24.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/5ff71989-ed8c-437b-bc33-5da799703f53_characterizeImage_1125541_24.stderr parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/be4652cc-b5cb-4127-9a29-d67ab4f0f2be_isr_1125541_24.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/be4652cc-b5cb-4127-9a29-d67ab4f0f2be_isr_1125541_24.stderr parsl.dataflow.dflow INFO: Task 297 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62b0a1702a230b9b6725e1666a9e26cb with result from task 297 parsl.dataflow.dflow DEBUG: Task 297 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 298 has memoization hash a0f29adcb47fb69ff6f6ea256dec343c parsl.dataflow.memoization INFO: Task 298 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f30b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 298 try 0 launched on executor multi with executor id 274 parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/dae85609-6d64-4adc-830b-410ea0b3c9ee_characterizeImage_1125543_31.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/dae85609-6d64-4adc-830b-410ea0b3c9ee_characterizeImage_1125543_31.stderr parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/e36b8eab-3b72-40f9-b459-58947a6f8c80_isr_1125543_31.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/e36b8eab-3b72-40f9-b459-58947a6f8c80_isr_1125543_31.stderr parsl.dataflow.dflow INFO: Task 294 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a80230efa790aead214b06c81106854 with result from task 294 parsl.dataflow.dflow DEBUG: Task 294 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 295 has memoization hash 5350aae8b892877fddd35e8b48614d06 parsl.dataflow.memoization INFO: Task 295 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f30860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 295 try 0 launched on executor multi with executor id 275 parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/82472f56-debe-400f-95e1-7b7fe56aea03_characterizeImage_1125541_14.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/82472f56-debe-400f-95e1-7b7fe56aea03_characterizeImage_1125541_14.stderr parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/e6078452-361b-4297-863f-5e5a1fc61790_isr_1125541_14.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/e6078452-361b-4297-863f-5e5a1fc61790_isr_1125541_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:23:47 proc_lsst.multi:146 [INFO] found job 16889845 in provider astro proc_lsst.multi INFO: found job 16889845 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:23:47 proc_lsst.multi:146 [INFO] found job 16889846 in provider astro proc_lsst.multi INFO: found job 16889846 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 300 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5c38299cac5cbc7416af00b0d5b7d5d with result from task 300 parsl.dataflow.dflow DEBUG: Task 300 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 301 has memoization hash 8829ccc8e3a2831c35a1770b5451c033 parsl.dataflow.memoization INFO: Task 301 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f30ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 301 try 0 launched on executor multi with executor id 276 parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/81432c41-a551-45bd-9a30-34db2866f125_characterizeImage_1125543_22.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/81432c41-a551-45bd-9a30-34db2866f125_characterizeImage_1125543_22.stderr parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/e4a5a323-7a8f-44ac-8dac-b63ce12afeeb_isr_1125543_22.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/e4a5a323-7a8f-44ac-8dac-b63ce12afeeb_isr_1125543_22.stderr parsl.dataflow.dflow INFO: Task 279 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e08803902ee658035c90cd5af303166 with result from task 279 parsl.dataflow.dflow DEBUG: Task 279 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 280 has memoization hash f1a8628a0929772a1e864656a33f395b parsl.dataflow.memoization INFO: Task 280 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eff880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 280 try 0 launched on executor multi with executor id 277 parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/7dc6a1a2-0c59-47c1-bef0-c1e84108dbc3_characterizeImage_1125543_54.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/7dc6a1a2-0c59-47c1-bef0-c1e84108dbc3_characterizeImage_1125543_54.stderr parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/504a48d6-e1a9-45e1-be6e-d14a0e871d27_isr_1125543_54.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/504a48d6-e1a9-45e1-be6e-d14a0e871d27_isr_1125543_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:23:48 proc_lsst.multi:146 [INFO] found job 16889847 in provider astro proc_lsst.multi INFO: found job 16889847 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:23:48 proc_lsst.multi:146 [INFO] found job 16889848 in provider astro proc_lsst.multi INFO: found job 16889848 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 276 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab3647035755ee6a84801aac26ce1901 with result from task 276 parsl.dataflow.dflow DEBUG: Task 276 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 277 has memoization hash fc4d6bc2babc79270769680177b47097 parsl.dataflow.memoization INFO: Task 277 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eff560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 277 try 0 launched on executor multi with executor id 278 parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/dc78f889-881c-4395-9f08-48866636df02_characterizeImage_1125541_1.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/dc78f889-881c-4395-9f08-48866636df02_characterizeImage_1125541_1.stderr parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/6d3455f2-4cff-4df7-920f-7799f3b39c5d_isr_1125541_1.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/6d3455f2-4cff-4df7-920f-7799f3b39c5d_isr_1125541_1.stderr parsl.dataflow.dflow INFO: Task 288 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f47be76591ca0a9d97f5777665131e4d with result from task 288 parsl.dataflow.dflow DEBUG: Task 288 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 289 has memoization hash b68834df984e8c379657dd061e71ef91 parsl.dataflow.memoization INFO: Task 289 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 289 try 0 launched on executor multi with executor id 279 parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/41fad328-bceb-4a92-a088-8e7e61452309_characterizeImage_1125542_53.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/41fad328-bceb-4a92-a088-8e7e61452309_characterizeImage_1125542_53.stderr parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/5095dd05-8431-45db-a081-055cf0e8757f_isr_1125542_53.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/5095dd05-8431-45db-a081-055cf0e8757f_isr_1125542_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:23:48 proc_lsst.multi:146 [INFO] found job 16889852 in provider astro proc_lsst.multi INFO: found job 16889852 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 267 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53dd1a0ecd45db616a1b1dcd4e003bf0 with result from task 267 parsl.dataflow.dflow DEBUG: Task 267 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 268 has memoization hash 0e9c8af4541d2b73e158d58bf1306a35 parsl.dataflow.memoization INFO: Task 268 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 268 try 0 launched on executor multi with executor id 280 parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/4b9e11f7-6e2f-4605-9a8b-4e084d8d0caa_characterizeImage_1125542_25.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/4b9e11f7-6e2f-4605-9a8b-4e084d8d0caa_characterizeImage_1125542_25.stderr parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/6d3bd26f-7a88-4440-b165-be27005649f5_isr_1125542_25.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/6d3bd26f-7a88-4440-b165-be27005649f5_isr_1125542_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:23:49 proc_lsst.multi:146 [INFO] found job 16889853 in provider astro proc_lsst.multi INFO: found job 16889853 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 258 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c87a71439dc8649cf1bb7bdb57a628b6 with result from task 258 parsl.dataflow.dflow DEBUG: Task 258 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 259 has memoization hash 1b0a5fd58c0c3a305c26eb82310d7542 parsl.dataflow.memoization INFO: Task 259 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efe2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 259 try 0 launched on executor multi with executor id 281 parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/91b32ed4-7ff0-45dc-813e-e245106f1baa_characterizeImage_1125543_58.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/91b32ed4-7ff0-45dc-813e-e245106f1baa_characterizeImage_1125543_58.stderr parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/00dee20c-ce45-4778-a2a5-81ac0b8e2ffc_isr_1125543_58.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/00dee20c-ce45-4778-a2a5-81ac0b8e2ffc_isr_1125543_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:23:49 proc_lsst.multi:146 [INFO] found job 16889854 in provider astro proc_lsst.multi INFO: found job 16889854 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 318 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e42316d03caaaa691283baeb57af35b9 with result from task 318 parsl.dataflow.dflow DEBUG: Task 318 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 319 has memoization hash f735f177d9a1a55671b2427c6d932807 parsl.dataflow.memoization INFO: Task 319 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f32160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 319 try 0 launched on executor multi with executor id 282 parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/f64684ff-da22-42a8-a535-7b229c4af21c_characterizeImage_1125541_31.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/f64684ff-da22-42a8-a535-7b229c4af21c_characterizeImage_1125541_31.stderr parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/56366522-7ce3-430e-9d6d-391766775429_isr_1125541_31.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/56366522-7ce3-430e-9d6d-391766775429_isr_1125541_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:23:49 proc_lsst.multi:146 [INFO] found job 16889855 in provider astro proc_lsst.multi INFO: found job 16889855 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 282 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0531573469ccddbd98dcdd5b5a244cdb with result from task 282 parsl.dataflow.dflow DEBUG: Task 282 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 283 has memoization hash a5aec5e8b19c9ed9a4537348f90355ba parsl.dataflow.memoization INFO: Task 283 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811effba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 283 try 0 launched on executor multi with executor id 283 parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/51869941-58e3-4045-ab43-1c9b2fa1b986_characterizeImage_1125543_7.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/51869941-58e3-4045-ab43-1c9b2fa1b986_characterizeImage_1125543_7.stderr parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/94e3ef5a-e56f-467e-8fc0-2c7844cd7036_isr_1125543_7.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/94e3ef5a-e56f-467e-8fc0-2c7844cd7036_isr_1125543_7.stderr parsl.dataflow.dflow INFO: Task 306 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd4c213c8515a3b663ba51f8e46ddbcb with result from task 306 parsl.dataflow.dflow DEBUG: Task 306 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 307 has memoization hash 5e26399a39d871f07f5b2bf02df8045d parsl.dataflow.memoization INFO: Task 307 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f314e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 307 try 0 launched on executor multi with executor id 284 parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/ac706922-c293-4d4e-9e01-b6914ac04119_characterizeImage_1125542_1.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/ac706922-c293-4d4e-9e01-b6914ac04119_characterizeImage_1125542_1.stderr parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/9d7488c3-4f16-4cc8-8bc6-972c454ffe2d_isr_1125542_1.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/9d7488c3-4f16-4cc8-8bc6-972c454ffe2d_isr_1125542_1.stderr parsl.dataflow.dflow INFO: Task 264 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb1dfad588d839aeb513c408ae3b00bb with result from task 264 parsl.dataflow.dflow DEBUG: Task 264 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 265 has memoization hash 18c0e625a89297d3afdb3bc430dbfaed parsl.dataflow.memoization INFO: Task 265 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efe8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 265 try 0 launched on executor multi with executor id 285 parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/c70ccf96-444f-4fa7-93a4-da00461e04d7_characterizeImage_1125542_17.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/c70ccf96-444f-4fa7-93a4-da00461e04d7_characterizeImage_1125542_17.stderr parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/cfce0014-14bd-4e3f-994f-2774c53561cf_isr_1125542_17.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/cfce0014-14bd-4e3f-994f-2774c53561cf_isr_1125542_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:23:50 proc_lsst.multi:146 [INFO] found job 16889856 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889856 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.dataflow.dflow INFO: Task 321 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a06f724f54c2600770aaf6712503515 with result from task 321 parsl.dataflow.dflow DEBUG: Task 321 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 322 has memoization hash ead9ba522a6fdc834759310cb29ad72f parsl.dataflow.memoization INFO: Task 322 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f32480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 322 try 0 launched on executor multi with executor id 286 parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/262ccbcd-d805-4531-ad67-0bf24c899881_characterizeImage_1125541_36.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/262ccbcd-d805-4531-ad67-0bf24c899881_characterizeImage_1125541_36.stderr parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/d45aeff2-c5f3-4897-9fd3-dd5478dac7c9_isr_1125541_36.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/d45aeff2-c5f3-4897-9fd3-dd5478dac7c9_isr_1125541_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:23:52 proc_lsst.multi:146 [INFO] found job 16889857 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889857 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:23:52 proc_lsst.multi:146 [INFO] found job 16889858 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889858 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:23:53 proc_lsst.multi:146 [INFO] found job 16889859 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889859 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:23:53 proc_lsst.multi:146 [INFO] found job 16889860 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889860 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:23:54 proc_lsst.multi:146 [INFO] found job 16889861 in provider escience_a40 proc_lsst.multi INFO: found job 16889861 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 342 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f701bd84287da24567a320c6deca720 with result from task 342 parsl.dataflow.dflow DEBUG: Task 342 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 343 has memoization hash 320de90e2b1ff9d8d663fb0bef25ae7a parsl.dataflow.memoization INFO: Task 343 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f33a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 343 try 0 launched on executor multi with executor id 287 parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/1baf5726-d1b3-44f1-9fa3-6a83de5ab826_characterizeImage_1125541_19.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/1baf5726-d1b3-44f1-9fa3-6a83de5ab826_characterizeImage_1125541_19.stderr parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/0626cc5a-73df-464a-9c9d-af99a6cae287_isr_1125541_19.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/0626cc5a-73df-464a-9c9d-af99a6cae287_isr_1125541_19.stderr parsl.dataflow.dflow INFO: Task 327 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8ada7cb7908e0a7533b396db5c6c778 with result from task 327 parsl.dataflow.dflow DEBUG: Task 327 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 328 has memoization hash f4aff324b0b49346d5ee18d13bf216dd parsl.dataflow.memoization INFO: Task 328 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f32ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 328 try 0 launched on executor multi with executor id 288 parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/b2878146-f284-4dc5-a2e5-b525e5c594dc_characterizeImage_1125541_41.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/b2878146-f284-4dc5-a2e5-b525e5c594dc_characterizeImage_1125541_41.stderr parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/21c7fc55-e256-46c0-912f-a8e4c16e96c1_isr_1125541_41.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/21c7fc55-e256-46c0-912f-a8e4c16e96c1_isr_1125541_41.stderr parsl.dataflow.dflow INFO: Task 324 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9899a7f95e68afbd7d21da433dbcd12 with result from task 324 parsl.dataflow.dflow DEBUG: Task 324 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 325 has memoization hash bcbb9d6d514cfc7d16d6b0b64539ce22 parsl.dataflow.memoization INFO: Task 325 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f327a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 325 try 0 launched on executor multi with executor id 289 parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/b274198d-ef9f-44d5-b778-c264ecf488e5_characterizeImage_1125542_36.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/b274198d-ef9f-44d5-b778-c264ecf488e5_characterizeImage_1125542_36.stderr parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/62171297-995f-4dfe-a47b-7d52417e64b9_isr_1125542_36.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/62171297-995f-4dfe-a47b-7d52417e64b9_isr_1125542_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:23:54 proc_lsst.multi:146 [INFO] found job 16889862 in provider escience_a40 proc_lsst.multi INFO: found job 16889862 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 336 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fd7664c112029aa0daf7786b74950d7 with result from task 336 parsl.dataflow.dflow DEBUG: Task 336 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 337 has memoization hash f0a27f2bdc0f624ebf49c31d11777736 parsl.dataflow.memoization INFO: Task 337 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f33420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 337 try 0 launched on executor multi with executor id 290 parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/f7afc55c-ed96-43b9-9dcf-f9c5c202fefc_characterizeImage_1125543_3.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/f7afc55c-ed96-43b9-9dcf-f9c5c202fefc_characterizeImage_1125543_3.stderr parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/12b3fa96-2f81-45de-a6a6-609f0c4faddf_isr_1125543_3.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/12b3fa96-2f81-45de-a6a6-609f0c4faddf_isr_1125543_3.stderr parsl.dataflow.dflow INFO: Task 357 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 227251d1ca6253dbf3ae0ce45998adca with result from task 357 parsl.dataflow.dflow DEBUG: Task 357 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 358 has memoization hash cd3f49f6622a4bba8663f9acbbfbe2fa parsl.dataflow.memoization INFO: Task 358 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f78a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 358 try 0 launched on executor multi with executor id 291 parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/d6a8aed5-11ca-48a1-b29a-f05fc8f92d46_characterizeImage_1125542_47.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/d6a8aed5-11ca-48a1-b29a-f05fc8f92d46_characterizeImage_1125542_47.stderr parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/617c1c4f-f657-4aed-a863-855cfec9a587_isr_1125542_47.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/617c1c4f-f657-4aed-a863-855cfec9a587_isr_1125542_47.stderr parsl.dataflow.dflow INFO: Task 330 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6768d8a281149117e3c836b828cbed5 with result from task 330 parsl.dataflow.dflow DEBUG: Task 330 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 331 has memoization hash 14b19c4a849405ae3ba140fb8cb0a810 parsl.dataflow.memoization INFO: Task 331 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f32de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 331 try 0 launched on executor multi with executor id 292 parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/f636359e-3add-4cf2-83b2-45514b25c8d5_characterizeImage_1125542_62.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/f636359e-3add-4cf2-83b2-45514b25c8d5_characterizeImage_1125542_62.stderr parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/9137e1f1-2b8a-467d-93a4-44f49fd640c3_isr_1125542_62.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/9137e1f1-2b8a-467d-93a4-44f49fd640c3_isr_1125542_62.stderr parsl.dataflow.dflow INFO: Task 354 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94540944bfc0e38a342487f150fb5d3d with result from task 354 parsl.dataflow.dflow DEBUG: Task 354 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 355 has memoization hash c8b2306d8ee054dee80c9d11f780febb parsl.dataflow.memoization INFO: Task 355 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f78720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 355 try 0 launched on executor multi with executor id 293 parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/35adb483-89a5-4584-9817-8733569d4390_characterizeImage_1125542_11.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/35adb483-89a5-4584-9817-8733569d4390_characterizeImage_1125542_11.stderr parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/68241d3d-257e-494c-bc5f-08580d2eba8c_isr_1125542_11.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/68241d3d-257e-494c-bc5f-08580d2eba8c_isr_1125542_11.stderr parsl.dataflow.dflow INFO: Task 360 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc894d26f4fff22ef97231b6e888ebeb with result from task 360 parsl.dataflow.dflow DEBUG: Task 360 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 361 has memoization hash 0891dbc7bdd25784d2229be6ce8cf3e0 parsl.dataflow.memoization INFO: Task 361 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f78d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 361 try 0 launched on executor multi with executor id 294 parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/dfcfdc1e-a07d-4168-b007-bc0c87754827_characterizeImage_1125543_10.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/dfcfdc1e-a07d-4168-b007-bc0c87754827_characterizeImage_1125543_10.stderr parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/e4a93610-5011-4537-ace1-bddf13e49b17_isr_1125543_10.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/e4a93610-5011-4537-ace1-bddf13e49b17_isr_1125543_10.stderr parsl.dataflow.dflow INFO: Task 333 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 653a1793375140398670707e7c95a234 with result from task 333 parsl.dataflow.dflow DEBUG: Task 333 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 334 has memoization hash a5055693394285aedeb36964006cce3d parsl.dataflow.memoization INFO: Task 334 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f33100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 334 try 0 launched on executor multi with executor id 295 parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/0ef28c78-8097-4565-b312-36f54e69391c_characterizeImage_1125542_18.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/0ef28c78-8097-4565-b312-36f54e69391c_characterizeImage_1125542_18.stderr parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/f10cf8dd-785e-4900-8728-2570425b6c97_isr_1125542_18.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/f10cf8dd-785e-4900-8728-2570425b6c97_isr_1125542_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:23:55 proc_lsst.multi:146 [INFO] found job 16889863 in provider escience_a40 proc_lsst.multi INFO: found job 16889863 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 384 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d954fa987884b599b42b079ae38ff536 with result from task 384 parsl.dataflow.dflow DEBUG: Task 384 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 385 has memoization hash 7004952a6f1c745e125807ebc5b5fc7d parsl.dataflow.memoization INFO: Task 385 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f7a660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 385 try 0 launched on executor multi with executor id 296 parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/bf4b57dd-7fb3-4e1b-a196-4164dfe425fc_characterizeImage_1125542_57.stdout parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/bf4b57dd-7fb3-4e1b-a196-4164dfe425fc_characterizeImage_1125542_57.stderr parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/49e2b844-02cd-4232-9605-c82a179bb3c5_isr_1125542_57.stdout parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/49e2b844-02cd-4232-9605-c82a179bb3c5_isr_1125542_57.stderr parsl.dataflow.dflow INFO: Task 375 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6ce1639b3532ac99d68ce0e56bbb736 with result from task 375 parsl.dataflow.dflow DEBUG: Task 375 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 376 has memoization hash 3381cac2f62a77db434de542e9734da9 parsl.dataflow.memoization INFO: Task 376 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f79d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 376 try 0 launched on executor multi with executor id 297 parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/f12171ac-6d29-4b1a-b40c-67b9658c5d34_characterizeImage_1125541_22.stdout parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/f12171ac-6d29-4b1a-b40c-67b9658c5d34_characterizeImage_1125541_22.stderr parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/3b58dfc5-c188-47fa-a1fa-8b53cd718d55_isr_1125541_22.stdout parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/3b58dfc5-c188-47fa-a1fa-8b53cd718d55_isr_1125541_22.stderr parsl.dataflow.dflow INFO: Task 351 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 226de6ede28c9e0cb14af37d4f2b52d9 with result from task 351 parsl.dataflow.dflow DEBUG: Task 351 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 352 has memoization hash 9485271fe8568fe698882d79a86e3040 parsl.dataflow.memoization INFO: Task 352 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f78400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 352 try 0 launched on executor multi with executor id 298 parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/82f41c71-8d8d-4b5f-9a34-43467ee84bd2_characterizeImage_1125543_55.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/82f41c71-8d8d-4b5f-9a34-43467ee84bd2_characterizeImage_1125543_55.stderr parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/193b50d2-77f5-4ea8-ad6d-6f7fb23c67b3_isr_1125543_55.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/193b50d2-77f5-4ea8-ad6d-6f7fb23c67b3_isr_1125543_55.stderr parsl.dataflow.dflow INFO: Task 348 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5493812eb6a6c68a1e07c77504359c44 with result from task 348 parsl.dataflow.dflow DEBUG: Task 348 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 349 has memoization hash 30cf5bb14acf3c2b4f27f7b423d3a17c parsl.dataflow.memoization INFO: Task 349 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f33ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 349 try 0 launched on executor multi with executor id 299 parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/5f678f28-605d-4aec-9ff8-d4b815838c62_characterizeImage_1125542_37.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/5f678f28-605d-4aec-9ff8-d4b815838c62_characterizeImage_1125542_37.stderr parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/ba2f6f06-929a-44cd-bc82-599c60852e4f_isr_1125542_37.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/ba2f6f06-929a-44cd-bc82-599c60852e4f_isr_1125542_37.stderr parsl.dataflow.dflow INFO: Task 396 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d795c30d52e0bde95b22dd8ac3256180 with result from task 396 parsl.dataflow.dflow DEBUG: Task 396 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 397 has memoization hash 17328f14d533e9b2da0e96d7d20aee48 parsl.dataflow.memoization INFO: Task 397 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbdd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 397 try 0 launched on executor multi with executor id 300 parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/4f8ca024-7872-4a1a-bfd1-9c4999e35cff_characterizeImage_1125543_33.stdout parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/4f8ca024-7872-4a1a-bfd1-9c4999e35cff_characterizeImage_1125543_33.stderr parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/7cfbfbcb-b50e-404e-8e18-744a981492f9_isr_1125543_33.stdout parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/7cfbfbcb-b50e-404e-8e18-744a981492f9_isr_1125543_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:23:56 proc_lsst.multi:146 [INFO] found job 16889864 in provider escience_a40 proc_lsst.multi INFO: found job 16889864 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 315 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c43f48ad74d4aecc40948ae61e18e28f with result from task 315 parsl.dataflow.dflow DEBUG: Task 315 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 316 has memoization hash a2f88140a94ac27069f6de84236d02c0 parsl.dataflow.memoization INFO: Task 316 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f31e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 316 try 0 launched on executor multi with executor id 301 parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/b0ad3336-2037-46ae-9f77-d21e3f611026_characterizeImage_1125543_37.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/b0ad3336-2037-46ae-9f77-d21e3f611026_characterizeImage_1125543_37.stderr parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/0b8648c0-cdb8-4ef7-9da8-e344ca9724f1_isr_1125543_37.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/0b8648c0-cdb8-4ef7-9da8-e344ca9724f1_isr_1125543_37.stderr parsl.dataflow.dflow INFO: Task 378 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f93085d0d9eb82163d3e73a2091cb56e with result from task 378 parsl.dataflow.dflow DEBUG: Task 378 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 379 has memoization hash 69a5b7c8c1070b4151d2b0b839743fb9 parsl.dataflow.memoization INFO: Task 379 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f7a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 379 try 0 launched on executor multi with executor id 302 parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/34b82464-82b0-4450-8b12-c4ca7a5e2ee7_characterizeImage_1125543_30.stdout parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/34b82464-82b0-4450-8b12-c4ca7a5e2ee7_characterizeImage_1125543_30.stderr parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/4e95d249-6fe4-4f28-8409-8a48c8ff23e6_isr_1125543_30.stdout parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/4e95d249-6fe4-4f28-8409-8a48c8ff23e6_isr_1125543_30.stderr parsl.dataflow.dflow INFO: Task 390 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9aa6751a9144d34a7d29fdf71823840f with result from task 390 parsl.dataflow.dflow DEBUG: Task 390 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 391 has memoization hash c060318160e09e93f3a8583e151b0315 parsl.dataflow.memoization INFO: Task 391 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f7aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 391 try 0 launched on executor multi with executor id 303 parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/9eb275eb-1045-439f-868c-8588a45b81ce_characterizeImage_1125543_23.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/9eb275eb-1045-439f-868c-8588a45b81ce_characterizeImage_1125543_23.stderr parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/9577b3f2-a81b-4aab-9e96-6afc5f8cdffa_isr_1125543_23.stdout parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/9577b3f2-a81b-4aab-9e96-6afc5f8cdffa_isr_1125543_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:23:59 proc_lsst.multi:146 [INFO] found job 16889865 in provider escience_a40 proc_lsst.multi INFO: found job 16889865 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 399 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ea6f5236b8fb880442d1f80ca2a5432 with result from task 399 parsl.dataflow.dflow DEBUG: Task 399 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 400 has memoization hash c6723bb06593b7271777c7f1e55cc583 parsl.dataflow.memoization INFO: Task 400 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbd9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 400 try 0 launched on executor multi with executor id 304 parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/45c9f29a-ac2b-479b-a66a-1e9079f3bcc5_characterizeImage_1125541_54.stdout parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/45c9f29a-ac2b-479b-a66a-1e9079f3bcc5_characterizeImage_1125541_54.stderr parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/88721219-f55a-4047-855e-14d58b59c332_isr_1125541_54.stdout parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/88721219-f55a-4047-855e-14d58b59c332_isr_1125541_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:00 proc_lsst.multi:146 [INFO] found job 16889866 in provider escience_a40 proc_lsst.multi INFO: found job 16889866 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:01 proc_lsst.multi:146 [INFO] found job 16889867 in provider escience_a40 proc_lsst.multi INFO: found job 16889867 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 429 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54e461f67dc3e9d62c0a66945ae98c50 with result from task 429 parsl.dataflow.dflow DEBUG: Task 429 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 430 has memoization hash 4e6274f759f38c06f398425c15cb37be parsl.dataflow.memoization INFO: Task 430 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbc180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 430 try 0 launched on executor multi with executor id 305 parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/2c0df08f-9946-4340-bad2-a667913ea98d_characterizeImage_1125541_23.stdout parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/2c0df08f-9946-4340-bad2-a667913ea98d_characterizeImage_1125541_23.stderr parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/f0ccde67-2183-4b9a-b6a0-c2f6297b68c6_isr_1125541_23.stdout parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/f0ccde67-2183-4b9a-b6a0-c2f6297b68c6_isr_1125541_23.stderr parsl.dataflow.dflow INFO: Task 363 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de7242f41ce8aab377aa4c6717c6f645 with result from task 363 parsl.dataflow.dflow DEBUG: Task 363 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 364 has memoization hash 8ac4c763e62a64cb76cebb6ecb86a067 parsl.dataflow.memoization INFO: Task 364 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f79080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 364 try 0 launched on executor multi with executor id 306 parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/fe37ebae-12e4-4b18-a319-ab390078232f_characterizeImage_1125541_57.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/fe37ebae-12e4-4b18-a319-ab390078232f_characterizeImage_1125541_57.stderr parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/b1951421-e7ab-402e-97c0-68ef5ccc237e_isr_1125541_57.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/b1951421-e7ab-402e-97c0-68ef5ccc237e_isr_1125541_57.stderr parsl.dataflow.dflow INFO: Task 411 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 157ed4bb7ea40e87065c5da81020f457 with result from task 411 parsl.dataflow.dflow DEBUG: Task 411 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 412 has memoization hash 64550ef3cefc01039ea952de1e79fe31 parsl.dataflow.memoization INFO: Task 412 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f7af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 412 try 0 launched on executor multi with executor id 307 parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/5c5674d0-e8b2-453e-ba3a-2cd57655d869_characterizeImage_1125541_39.stdout parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/5c5674d0-e8b2-453e-ba3a-2cd57655d869_characterizeImage_1125541_39.stderr parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/ff368ccb-8905-40f7-a12d-78ae0eb6c5a6_isr_1125541_39.stdout parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/ff368ccb-8905-40f7-a12d-78ae0eb6c5a6_isr_1125541_39.stderr parsl.dataflow.dflow INFO: Task 372 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7613ea47e8a2ef0ec5b59cfa6529d42b with result from task 372 parsl.dataflow.dflow DEBUG: Task 372 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 373 has memoization hash e2e54ed358fd2348c77eaa49b9991b4e parsl.dataflow.memoization INFO: Task 373 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f799e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 373 try 0 launched on executor multi with executor id 308 parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/7cb1c164-33f5-45e7-a4c7-70629b295801_characterizeImage_1125542_39.stdout parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/7cb1c164-33f5-45e7-a4c7-70629b295801_characterizeImage_1125542_39.stderr parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/37cc7f84-ed18-4103-9f7d-d9b3f9c51504_isr_1125542_39.stdout parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/37cc7f84-ed18-4103-9f7d-d9b3f9c51504_isr_1125542_39.stderr parsl.dataflow.dflow INFO: Task 381 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64d10e5a7cb1b9ea5fcf086a2b2832f7 with result from task 381 parsl.dataflow.dflow DEBUG: Task 381 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 382 has memoization hash efa36736e35a1649b209502126d8da41 parsl.dataflow.memoization INFO: Task 382 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f7a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 382 try 0 launched on executor multi with executor id 309 parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/c625f50a-6e21-4583-9c3a-71d130963aad_characterizeImage_1125542_5.stdout parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/c625f50a-6e21-4583-9c3a-71d130963aad_characterizeImage_1125542_5.stderr parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/2a9f0580-2675-4f0f-99ca-82b2a3786cda_isr_1125542_5.stdout parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/2a9f0580-2675-4f0f-99ca-82b2a3786cda_isr_1125542_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:02 proc_lsst.multi:146 [INFO] found job 16889868 in provider escience_a40 proc_lsst.multi INFO: found job 16889868 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 366 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cca4d2ded55e99199edc51f4adb08b03 with result from task 366 parsl.dataflow.dflow DEBUG: Task 366 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 367 has memoization hash fdec887d2aac35275d3b075fc8ce9945 parsl.dataflow.memoization INFO: Task 367 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f793a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 367 try 0 launched on executor multi with executor id 310 parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/b1b96af1-ed22-4a53-8654-1f6010a5d895_characterizeImage_1125543_5.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/b1b96af1-ed22-4a53-8654-1f6010a5d895_characterizeImage_1125543_5.stderr parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/9f5664d4-7109-480d-866e-6b9e1cb7a48e_isr_1125543_5.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/9f5664d4-7109-480d-866e-6b9e1cb7a48e_isr_1125543_5.stderr parsl.dataflow.dflow INFO: Task 405 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4f3a3560b8d7f969d757f67d7a96331 with result from task 405 parsl.dataflow.dflow DEBUG: Task 405 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 406 has memoization hash eda4f16cc74127de328f775f893d2169 parsl.dataflow.memoization INFO: Task 406 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbd3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 406 try 0 launched on executor multi with executor id 311 parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/c8c937fa-8606-407b-a2a6-497f9e0815ea_characterizeImage_1125541_16.stdout parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/c8c937fa-8606-407b-a2a6-497f9e0815ea_characterizeImage_1125541_16.stderr parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/ee753b55-8e1f-4156-a346-3aabcb3b72ae_isr_1125541_16.stdout parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/ee753b55-8e1f-4156-a346-3aabcb3b72ae_isr_1125541_16.stderr parsl.dataflow.dflow INFO: Task 339 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d19730d789c245f9b2118994ef0f4e76 with result from task 339 parsl.dataflow.dflow DEBUG: Task 339 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 340 has memoization hash ecff4f05f5b7402097dcb6392dda78eb parsl.dataflow.memoization INFO: Task 340 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f33740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 340 try 0 launched on executor multi with executor id 312 parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/3d8e8ea3-f7a1-4f55-9417-564a6b799d5a_characterizeImage_1125543_28.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/3d8e8ea3-f7a1-4f55-9417-564a6b799d5a_characterizeImage_1125543_28.stderr parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/c82525f8-b079-4f59-bc1a-2f41367611da_isr_1125543_28.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/c82525f8-b079-4f59-bc1a-2f41367611da_isr_1125543_28.stderr parsl.dataflow.dflow INFO: Task 420 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1219ec330e731ae3129123d8b97824d9 with result from task 420 parsl.dataflow.dflow DEBUG: Task 420 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 421 has memoization hash bff74bed3f9b2167394fc251c4a9465e parsl.dataflow.memoization INFO: Task 421 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f7bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 421 try 0 launched on executor multi with executor id 313 parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/b87ae1b3-5089-43d9-a7e2-70db97c1e76e_characterizeImage_1125542_8.stdout parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/b87ae1b3-5089-43d9-a7e2-70db97c1e76e_characterizeImage_1125542_8.stderr parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/6d914876-b09f-4a88-92d5-dbc2b2f9615e_isr_1125542_8.stdout parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/6d914876-b09f-4a88-92d5-dbc2b2f9615e_isr_1125542_8.stderr parsl.dataflow.dflow INFO: Task 387 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2eb3cfca1c32e103a9e67dc2237bad1d with result from task 387 parsl.dataflow.dflow DEBUG: Task 387 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 388 has memoization hash 6615c80477cf5d80918311a7ba2113ee parsl.dataflow.memoization INFO: Task 388 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f7a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 388 try 0 launched on executor multi with executor id 314 parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/799012a6-a874-4e9c-b41e-160c5dfbb2ea_characterizeImage_1125541_47.stdout parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/799012a6-a874-4e9c-b41e-160c5dfbb2ea_characterizeImage_1125541_47.stderr parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/10db20cf-0824-4eaa-96fa-5e83db321814_isr_1125541_47.stdout parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/10db20cf-0824-4eaa-96fa-5e83db321814_isr_1125541_47.stderr parsl.dataflow.dflow INFO: Task 408 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fa3bb81fc4e9d98a94e4308d2342efd with result from task 408 parsl.dataflow.dflow DEBUG: Task 408 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 409 has memoization hash 2bab56a15b404d37584bcf968a0653a2 parsl.dataflow.memoization INFO: Task 409 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbd080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 409 try 0 launched on executor multi with executor id 315 parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/7bb1e95e-6059-46e9-85e9-5d958500a594_characterizeImage_1125543_1.stdout parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/7bb1e95e-6059-46e9-85e9-5d958500a594_characterizeImage_1125543_1.stderr parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/61a61bbd-cd56-41f3-90ba-42969c0e3dee_isr_1125543_1.stdout parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/61a61bbd-cd56-41f3-90ba-42969c0e3dee_isr_1125543_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:03 proc_lsst.multi:146 [INFO] found job 16889869 in provider escience_a40 proc_lsst.multi INFO: found job 16889869 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 414 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1108608fda7c1f6eb3fdf9b8cc184409 with result from task 414 parsl.dataflow.dflow DEBUG: Task 414 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 415 has memoization hash 4d2b0440dd73dc4728d1d02402dd2dac parsl.dataflow.memoization INFO: Task 415 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f7b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 415 try 0 launched on executor multi with executor id 316 parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/251a135e-d94e-4393-8da9-814b247bb8a8_characterizeImage_1125542_55.stdout parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/251a135e-d94e-4393-8da9-814b247bb8a8_characterizeImage_1125542_55.stderr parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/99994388-1301-450d-84d5-fd2b9b33ed12_isr_1125542_55.stdout parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/99994388-1301-450d-84d5-fd2b9b33ed12_isr_1125542_55.stderr parsl.dataflow.dflow INFO: Task 369 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7119c25b9fa2a97c7e9b08fb4fcc503 with result from task 369 parsl.dataflow.dflow DEBUG: Task 369 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 370 has memoization hash e6888aaa228551b4842b2c09c7ebbf56 parsl.dataflow.memoization INFO: Task 370 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f796c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 370 try 0 launched on executor multi with executor id 317 parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/834014b0-d80b-493b-8b20-2b77742cfd2f_characterizeImage_1125541_53.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/834014b0-d80b-493b-8b20-2b77742cfd2f_characterizeImage_1125541_53.stderr parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/69861417-d3c6-43d9-bbea-c896fdebb63a_isr_1125541_53.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/69861417-d3c6-43d9-bbea-c896fdebb63a_isr_1125541_53.stderr parsl.dataflow.dflow INFO: Task 402 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 225014778466dc89fa465452734e4e54 with result from task 402 parsl.dataflow.dflow DEBUG: Task 402 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 403 has memoization hash 37ac5b112cfa18deb3b681bcb77e00c9 parsl.dataflow.memoization INFO: Task 403 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbd6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 403 try 0 launched on executor multi with executor id 318 parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/112b936d-6d78-432e-a208-07e90a433e52_characterizeImage_1125542_59.stdout parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/112b936d-6d78-432e-a208-07e90a433e52_characterizeImage_1125542_59.stderr parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/4708e662-9fd3-4868-b489-a55dac48e23d_isr_1125542_59.stdout parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/4708e662-9fd3-4868-b489-a55dac48e23d_isr_1125542_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:03 proc_lsst.multi:146 [INFO] found job 16889870 in provider escience_a40 proc_lsst.multi INFO: found job 16889870 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 393 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26b7a57150073ee92bf1f039c91f2bc1 with result from task 393 parsl.dataflow.dflow DEBUG: Task 393 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 394 has memoization hash 77e35c5f0d9d802b51cd1abc9fe2a85a parsl.dataflow.memoization INFO: Task 394 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbe2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 394 try 0 launched on executor multi with executor id 319 parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/23f7349a-46cf-49a3-b08e-741c8a2bb8fe_characterizeImage_1125543_29.stdout parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/23f7349a-46cf-49a3-b08e-741c8a2bb8fe_characterizeImage_1125543_29.stderr parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/df732600-76e5-4a01-a20f-870b46a54542_isr_1125543_29.stdout parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/df732600-76e5-4a01-a20f-870b46a54542_isr_1125543_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:03 proc_lsst.multi:146 [INFO] found job 16889871 in provider escience_a40 proc_lsst.multi INFO: found job 16889871 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 441 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b532fba7645212d132ed1dbb914330e with result from task 441 parsl.dataflow.dflow DEBUG: Task 441 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 442 has memoization hash bc7c53a25e34f673a1e4add49aa12bf3 parsl.dataflow.memoization INFO: Task 442 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbe340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 442 try 0 launched on executor multi with executor id 320 parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/3aabb4da-7ee4-4f82-8c99-490629014bcb_characterizeImage_1125541_45.stdout parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/3aabb4da-7ee4-4f82-8c99-490629014bcb_characterizeImage_1125541_45.stderr parsl.dataflow.dflow INFO: Standard output for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/3af5122a-51f2-4ae6-a6af-57ee8100eefd_isr_1125541_45.stdout parsl.dataflow.dflow INFO: Standard error for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/3af5122a-51f2-4ae6-a6af-57ee8100eefd_isr_1125541_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22643663051856 2024-03-07 10:24:04 proc_lsst.multi:146 [INFO] found job 24403 in provider local proc_lsst.multi INFO: found job 24403 in provider local 2024-03-07 10:24:04 proc_lsst.multi:146 [INFO] found job 16889823 in provider astro proc_lsst.multi INFO: found job 16889823 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 345 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b31d2aa9a2b804da8496b2ac1bd3a83f with result from task 345 parsl.dataflow.dflow DEBUG: Task 345 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 346 has memoization hash 0a12515cf4df68a64c3a910b1db0914f parsl.dataflow.memoization INFO: Task 346 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f33d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 346 try 0 launched on executor multi with executor id 321 parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/bccca5bb-37d3-46f0-bcab-d18ed8e2e682_characterizeImage_1125541_3.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/bccca5bb-37d3-46f0-bcab-d18ed8e2e682_characterizeImage_1125541_3.stderr parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/c210bd7c-a407-4c16-96fc-7853a48fc3ff_isr_1125541_3.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/c210bd7c-a407-4c16-96fc-7853a48fc3ff_isr_1125541_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:04 proc_lsst.multi:146 [INFO] found job 16889824 in provider astro proc_lsst.multi INFO: found job 16889824 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:05 proc_lsst.multi:146 [INFO] found job 16889825 in provider astro proc_lsst.multi INFO: found job 16889825 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:05 proc_lsst.multi:146 [INFO] found job 16889826 in provider astro proc_lsst.multi INFO: found job 16889826 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:05 proc_lsst.multi:146 [INFO] found job 16889827 in provider astro proc_lsst.multi INFO: found job 16889827 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 432 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 843175779a023c188b3bb572eabc9fab with result from task 432 parsl.dataflow.dflow DEBUG: Task 432 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 433 has memoization hash b14316c4be2567381868c5d146574c96 parsl.dataflow.memoization INFO: Task 433 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbc4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 433 try 0 launched on executor multi with executor id 322 parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/7a64743c-eb25-4915-ad46-b65b1ad3fb80_characterizeImage_1125542_45.stdout parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/7a64743c-eb25-4915-ad46-b65b1ad3fb80_characterizeImage_1125542_45.stderr parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/5bd43068-0d75-4ee6-b147-e42cfd55ceae_isr_1125542_45.stdout parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/5bd43068-0d75-4ee6-b147-e42cfd55ceae_isr_1125542_45.stderr parsl.dataflow.dflow INFO: Task 417 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65f6d8e23894512675d37cfd339163a5 with result from task 417 parsl.dataflow.dflow DEBUG: Task 417 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 418 has memoization hash e56ef9e565b53fef1af5e79b6527efcf parsl.dataflow.memoization INFO: Task 418 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f7b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:05 proc_lsst.multi:146 [INFO] found job 16889828 in provider astro proc_lsst.multi INFO: found job 16889828 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Parsl task 418 try 0 launched on executor multi with executor id 323 parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/7fb8c21b-e30b-4953-8e2d-90aec3561689_characterizeImage_1125542_3.stdout parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/7fb8c21b-e30b-4953-8e2d-90aec3561689_characterizeImage_1125542_3.stderr parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/cf8614b2-dda7-4ced-928b-f4da04138605_isr_1125542_3.stdout parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/cf8614b2-dda7-4ced-928b-f4da04138605_isr_1125542_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:06 proc_lsst.multi:146 [INFO] found job 16889829 in provider astro proc_lsst.multi INFO: found job 16889829 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 426 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d72e7d96d2d9d631eac74c7f93b826c7 with result from task 426 parsl.dataflow.dflow DEBUG: Task 426 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 427 has memoization hash 69acdeff407f14eb8744ced7ad1e1f46 parsl.dataflow.memoization INFO: Task 427 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f7bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 427 try 0 launched on executor multi with executor id 324 parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/6ff603f2-6cd6-464e-b221-0b77dced7b8e_characterizeImage_1125541_60.stdout parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/6ff603f2-6cd6-464e-b221-0b77dced7b8e_characterizeImage_1125541_60.stderr parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/2f5c430d-36a6-4612-8a80-61daa8692d32_isr_1125541_60.stdout parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/2f5c430d-36a6-4612-8a80-61daa8692d32_isr_1125541_60.stderr parsl.dataflow.dflow INFO: Task 435 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83d5b0d537f819d9e5c7fbd19da78314 with result from task 435 parsl.dataflow.dflow DEBUG: Task 435 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 436 has memoization hash 414f006eb60a2e1521638e867a636ecb parsl.dataflow.memoization INFO: Task 436 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbc900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 436 try 0 launched on executor multi with executor id 325 parsl.dataflow.dflow INFO: Standard output for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/6746dff4-adb2-468f-b598-dc028d3a7d66_characterizeImage_1125543_45.stdout parsl.dataflow.dflow INFO: Standard error for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/6746dff4-adb2-468f-b598-dc028d3a7d66_characterizeImage_1125543_45.stderr parsl.dataflow.dflow INFO: Standard output for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/7eb5b213-8a74-4db7-97bc-eececbf20036_isr_1125543_45.stdout parsl.dataflow.dflow INFO: Standard error for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/7eb5b213-8a74-4db7-97bc-eececbf20036_isr_1125543_45.stderr parsl.dataflow.dflow INFO: Task 423 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bdbdac7ff259b5f557c46105f4532494 with result from task 423 parsl.dataflow.dflow DEBUG: Task 423 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 424 has memoization hash 38355a3d3240891dacc1b4ad30b9a394 parsl.dataflow.memoization INFO: Task 424 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f7be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 424 try 0 launched on executor multi with executor id 326 parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/ed3e276f-6608-43dd-82b7-20ae5e8b96b9_characterizeImage_1125543_46.stdout parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/ed3e276f-6608-43dd-82b7-20ae5e8b96b9_characterizeImage_1125543_46.stderr parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/4bf98154-e27e-42d9-b043-1ad71869def1_isr_1125543_46.stdout parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/4bf98154-e27e-42d9-b043-1ad71869def1_isr_1125543_46.stderr parsl.dataflow.dflow INFO: Task 462 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75986978e99b40adcb355ed87de31a4b with result from task 462 parsl.dataflow.dflow DEBUG: Task 462 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 463 has memoization hash 5b4f8f99ea0f7e6dceb8d02aa549da6a parsl.dataflow.memoization INFO: Task 463 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfe520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 463 try 0 launched on executor multi with executor id 327 parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/77060bad-dfbf-498e-8b90-56db5c4b4019_characterizeImage_1125543_21.stdout parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/77060bad-dfbf-498e-8b90-56db5c4b4019_characterizeImage_1125543_21.stderr parsl.dataflow.dflow INFO: Standard output for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/c24b14a3-9513-46f8-b68f-d0018b9c0693_isr_1125543_21.stdout parsl.dataflow.dflow INFO: Standard error for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/c24b14a3-9513-46f8-b68f-d0018b9c0693_isr_1125543_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:06 proc_lsst.multi:146 [INFO] found job 16889830 in provider astro proc_lsst.multi INFO: found job 16889830 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:06 proc_lsst.multi:146 [INFO] found job 16889831 in provider astro proc_lsst.multi INFO: found job 16889831 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:06 proc_lsst.multi:146 [INFO] found job 16889832 in provider astro proc_lsst.multi INFO: found job 16889832 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:07 proc_lsst.multi:146 [INFO] found job 16889833 in provider astro proc_lsst.multi INFO: found job 16889833 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 471 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d19c9267fe4d0ae70fd4c2797cdf9a13 with result from task 471 parsl.dataflow.dflow DEBUG: Task 471 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 472 has memoization hash ab33c25d6e487ec9495b3a6d02fd4291 parsl.dataflow.memoization INFO: Task 472 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 472 try 0 launched on executor multi with executor id 328 parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/283d78b9-dcab-4b02-be4b-411548279a01_characterizeImage_1125542_41.stdout parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/283d78b9-dcab-4b02-be4b-411548279a01_characterizeImage_1125542_41.stderr parsl.dataflow.dflow INFO: Standard output for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/3a0073c5-9595-4847-b49d-c7f2656f4471_isr_1125542_41.stdout parsl.dataflow.dflow INFO: Standard error for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/3a0073c5-9595-4847-b49d-c7f2656f4471_isr_1125542_41.stderr parsl.dataflow.dflow INFO: Task 444 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9b22a277fa1700db31d6bb3df4607fa with result from task 444 parsl.dataflow.dflow DEBUG: Task 444 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 445 has memoization hash 83d976af8638af01445f585550003e77 parsl.dataflow.memoization INFO: Task 445 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbe520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 445 try 0 launched on executor multi with executor id 329 parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/8deb6b88-a428-47b5-a5e0-da6f787cc2b7_characterizeImage_1125541_40.stdout parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/8deb6b88-a428-47b5-a5e0-da6f787cc2b7_characterizeImage_1125541_40.stderr parsl.dataflow.dflow INFO: Standard output for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/19502779-309c-46fd-8c3f-01bb30eee660_isr_1125541_40.stdout parsl.dataflow.dflow INFO: Standard error for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/19502779-309c-46fd-8c3f-01bb30eee660_isr_1125541_40.stderr parsl.dataflow.dflow INFO: Task 456 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f089afdcd152567796dfe91247d3a91 with result from task 456 parsl.dataflow.dflow DEBUG: Task 456 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 457 has memoization hash c51666a053c3dc137d53932a3a19e00a parsl.dataflow.memoization INFO: Task 457 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbf380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 457 try 0 launched on executor multi with executor id 330 parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/b02bc6cb-4693-47d2-967f-7f96ea189338_characterizeImage_1125541_34.stdout parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/b02bc6cb-4693-47d2-967f-7f96ea189338_characterizeImage_1125541_34.stderr parsl.dataflow.dflow INFO: Standard output for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/61320e9e-fd5b-483f-8149-4b882b666c71_isr_1125541_34.stdout parsl.dataflow.dflow INFO: Standard error for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/61320e9e-fd5b-483f-8149-4b882b666c71_isr_1125541_34.stderr parsl.dataflow.dflow INFO: Task 465 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b75bb89956717cadc3f86f8ba600f35 with result from task 465 parsl.dataflow.dflow DEBUG: Task 465 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 466 has memoization hash f5e8e5c5b3a2c25685fb0a9ec9646107 parsl.dataflow.memoization INFO: Task 466 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbf560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 466 try 0 launched on executor multi with executor id 331 parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/0eaca5d5-2ceb-4095-b1aa-649dc0117180_characterizeImage_1125543_49.stdout parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/0eaca5d5-2ceb-4095-b1aa-649dc0117180_characterizeImage_1125543_49.stderr parsl.dataflow.dflow INFO: Standard output for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/7df55f4c-9afb-4bd9-9895-b3ec1ca2bfc2_isr_1125543_49.stdout parsl.dataflow.dflow INFO: Standard error for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/7df55f4c-9afb-4bd9-9895-b3ec1ca2bfc2_isr_1125543_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:07 proc_lsst.multi:146 [INFO] found job 16889834 in provider astro proc_lsst.multi INFO: found job 16889834 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 438 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4aefb6cbdbc99741072bbbf0e0da7712 with result from task 438 parsl.dataflow.dflow DEBUG: Task 438 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 439 has memoization hash 9d307c01b55192a97ca92e2dd8ad5aab parsl.dataflow.memoization INFO: Task 439 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbcc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 439 try 0 launched on executor multi with executor id 332 parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/26871cf2-7dbb-4c7e-a21e-6c07a785a5ea_characterizeImage_1125543_61.stdout parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/26871cf2-7dbb-4c7e-a21e-6c07a785a5ea_characterizeImage_1125543_61.stderr parsl.dataflow.dflow INFO: Standard output for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/7dc8c25e-2a0d-49f7-a3a6-902cfb874637_isr_1125543_61.stdout parsl.dataflow.dflow INFO: Standard error for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/7dc8c25e-2a0d-49f7-a3a6-902cfb874637_isr_1125543_61.stderr parsl.dataflow.dflow INFO: Task 228 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6f1406e2fe5422ce5ce43d76a45224c with result from task 228 parsl.dataflow.dflow DEBUG: Task 228 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 229 has memoization hash 91e188b88878c8f1a6886bae82ae04b6 parsl.dataflow.memoization INFO: Task 229 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efc360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 229 try 0 launched on executor multi with executor id 333 parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/2cb660db-45e1-4d86-906d-48445abaa853_characterizeImage_1125542_12.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/2cb660db-45e1-4d86-906d-48445abaa853_characterizeImage_1125542_12.stderr parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/2f5add86-0d46-4e7d-b629-43b7f7f30f7e_isr_1125542_12.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/2f5add86-0d46-4e7d-b629-43b7f7f30f7e_isr_1125542_12.stderr parsl.dataflow.dflow INFO: Task 231 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c911b3d5cb40ab1075286576edc73d56 with result from task 231 parsl.dataflow.dflow DEBUG: Task 231 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 232 has memoization hash 21c9a1334b41cb4007c17f720561c14b parsl.dataflow.memoization INFO: Task 232 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efc680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 232 try 0 launched on executor multi with executor id 334 parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/5bf29d16-6476-437b-9284-1826186ac188_characterizeImage_1125543_14.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/5bf29d16-6476-437b-9284-1826186ac188_characterizeImage_1125543_14.stderr parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/05eb6f32-ed8a-430c-92f0-c355a7a62b28_isr_1125543_14.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/05eb6f32-ed8a-430c-92f0-c355a7a62b28_isr_1125543_14.stderr parsl.dataflow.dflow INFO: Task 225 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 694adb729cfc301813567e24677b3107 with result from task 225 parsl.dataflow.dflow DEBUG: Task 225 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 226 has memoization hash 7315fe536273316a6b38cbbc7d3e22a1 parsl.dataflow.memoization INFO: Task 226 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed7e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 226 try 0 launched on executor multi with executor id 335 parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/0dd4318d-571a-44d6-915e-e6ee04157ddf_characterizeImage_1125543_51.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/0dd4318d-571a-44d6-915e-e6ee04157ddf_characterizeImage_1125543_51.stderr parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/6e74b57b-a936-458c-acd6-78115344feee_isr_1125543_51.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/6e74b57b-a936-458c-acd6-78115344feee_isr_1125543_51.stderr parsl.dataflow.dflow INFO: Task 222 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b3dbe0aeed1b168fd03e84ddbbc4b9b with result from task 222 parsl.dataflow.dflow DEBUG: Task 222 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 223 has memoization hash 2fee2ced1baba9fd8a15e920029d11aa parsl.dataflow.memoization INFO: Task 223 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed7ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 223 try 0 launched on executor multi with executor id 336 parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/572582d7-e2f9-453d-be08-1033d912a776_characterizeImage_1125543_48.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/572582d7-e2f9-453d-be08-1033d912a776_characterizeImage_1125543_48.stderr parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/31ec9599-2a0b-4056-b619-ab8af8a4c089_isr_1125543_48.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/31ec9599-2a0b-4056-b619-ab8af8a4c089_isr_1125543_48.stderr parsl.dataflow.dflow INFO: Task 219 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e76ea1df35edd22b91bdac87337d8f9 with result from task 219 parsl.dataflow.dflow DEBUG: Task 219 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 220 has memoization hash fb2d3774fe303951b8e6bbb743ef42e4 parsl.dataflow.memoization INFO: Task 220 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed79c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 220 try 0 launched on executor multi with executor id 337 parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/169a6baa-89bc-435b-956e-f111c48c2894_characterizeImage_1125542_23.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/169a6baa-89bc-435b-956e-f111c48c2894_characterizeImage_1125542_23.stderr parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/2c3a7240-e51f-4d2d-b970-dc75a49c81cb_isr_1125542_23.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/2c3a7240-e51f-4d2d-b970-dc75a49c81cb_isr_1125542_23.stderr parsl.dataflow.dflow INFO: Task 234 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b86b17fd58ebf6e8cbba44e8241eff1b with result from task 234 parsl.dataflow.dflow DEBUG: Task 234 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 235 has memoization hash e9abab7cb64f439e7eb9b78ef03ffc5f parsl.dataflow.memoization INFO: Task 235 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efc9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 235 try 0 launched on executor multi with executor id 338 parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/7b03af8e-7bbb-447c-aefa-29a65bd50a6c_characterizeImage_1125543_34.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/7b03af8e-7bbb-447c-aefa-29a65bd50a6c_characterizeImage_1125543_34.stderr parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/4365a311-8994-4e72-9dca-10d507522feb_isr_1125543_34.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/4365a311-8994-4e72-9dca-10d507522feb_isr_1125543_34.stderr parsl.dataflow.dflow INFO: Task 237 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0472ecb74c3b4f072861b0a9f8892b5a with result from task 237 parsl.dataflow.dflow DEBUG: Task 237 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 238 has memoization hash 4d2635cc09059694628ffd5d2ab3ad4b parsl.dataflow.memoization INFO: Task 238 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 238 try 0 launched on executor multi with executor id 339 parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/259f35ef-f9c9-437c-a4bb-000d2b793a69_characterizeImage_1125543_4.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/259f35ef-f9c9-437c-a4bb-000d2b793a69_characterizeImage_1125543_4.stderr parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/effe8e41-50b4-4769-a9ad-4102c756300a_isr_1125543_4.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/effe8e41-50b4-4769-a9ad-4102c756300a_isr_1125543_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:07 proc_lsst.multi:146 [INFO] found job 16889835 in provider astro proc_lsst.multi INFO: found job 16889835 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 453 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 275e6552a19002d377a590ee0cf57a8e with result from task 453 parsl.dataflow.dflow DEBUG: Task 453 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 454 has memoization hash 08d5c88cff836d533e437aa4c04eba68 parsl.dataflow.memoization INFO: Task 454 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbf060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 454 try 0 launched on executor multi with executor id 340 parsl.dataflow.dflow INFO: Standard output for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/f253d20e-116a-431d-bca3-c385f0581b8f_characterizeImage_1125541_18.stdout parsl.dataflow.dflow INFO: Standard error for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/f253d20e-116a-431d-bca3-c385f0581b8f_characterizeImage_1125541_18.stderr parsl.dataflow.dflow INFO: Standard output for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/afad57de-6dfd-414d-bb45-3e5ae3ee6be4_isr_1125541_18.stdout parsl.dataflow.dflow INFO: Standard error for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/afad57de-6dfd-414d-bb45-3e5ae3ee6be4_isr_1125541_18.stderr parsl.dataflow.dflow INFO: Task 483 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13442686cdc5056e00311c50744183f2 with result from task 483 parsl.dataflow.dflow DEBUG: Task 483 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 484 has memoization hash e7b9987a1f23630a934b56b5d07e349f parsl.dataflow.memoization INFO: Task 484 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfc7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 484 try 0 launched on executor multi with executor id 341 parsl.dataflow.dflow INFO: Standard output for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/bc1ee523-d07a-4040-b03f-aa79d12946f7_characterizeImage_1125543_53.stdout parsl.dataflow.dflow INFO: Standard error for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/bc1ee523-d07a-4040-b03f-aa79d12946f7_characterizeImage_1125543_53.stderr parsl.dataflow.dflow INFO: Standard output for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/6d170fe3-7f39-4b34-af48-211af76a2d14_isr_1125543_53.stdout parsl.dataflow.dflow INFO: Standard error for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/6d170fe3-7f39-4b34-af48-211af76a2d14_isr_1125543_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:08 proc_lsst.multi:146 [INFO] found job 16889836 in provider astro proc_lsst.multi INFO: found job 16889836 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 450 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ce39a0f1f948ce1719d45f768f8857f with result from task 450 parsl.dataflow.dflow DEBUG: Task 450 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 451 has memoization hash 08296ae5e544f5351cf6356d35f5fff6 parsl.dataflow.memoization INFO: Task 451 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 451 try 0 launched on executor multi with executor id 342 parsl.dataflow.dflow INFO: Standard output for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/aa571e3e-cdd3-455e-aa11-e5916dce620e_characterizeImage_1125541_15.stdout parsl.dataflow.dflow INFO: Standard error for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/aa571e3e-cdd3-455e-aa11-e5916dce620e_characterizeImage_1125541_15.stderr parsl.dataflow.dflow INFO: Standard output for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/a0cb96bd-24ad-46aa-b8a4-6c9d7ade28ff_isr_1125541_15.stdout parsl.dataflow.dflow INFO: Standard error for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/a0cb96bd-24ad-46aa-b8a4-6c9d7ade28ff_isr_1125541_15.stderr parsl.dataflow.dflow INFO: Task 477 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ecb0b17bba91458dd2ea6f7947f03b0a with result from task 477 parsl.dataflow.dflow DEBUG: Task 477 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 478 has memoization hash 4f506f4331b80bfbbb2e6bfe759fddf2 parsl.dataflow.memoization INFO: Task 478 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfe160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 478 try 0 launched on executor multi with executor id 343 parsl.dataflow.dflow INFO: Standard output for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/be1aa591-2dee-4679-bc36-579a930c5ee2_characterizeImage_1125542_30.stdout parsl.dataflow.dflow INFO: Standard error for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/be1aa591-2dee-4679-bc36-579a930c5ee2_characterizeImage_1125542_30.stderr parsl.dataflow.dflow INFO: Standard output for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/1a346cc0-06b4-487b-87fd-cb5e8d3032db_isr_1125542_30.stdout parsl.dataflow.dflow INFO: Standard error for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/1a346cc0-06b4-487b-87fd-cb5e8d3032db_isr_1125542_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:08 proc_lsst.multi:146 [INFO] found job 16889837 in provider astro proc_lsst.multi INFO: found job 16889837 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 474 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bbbe9b617c9311689b3fd4f2d6ac80a9 with result from task 474 parsl.dataflow.dflow DEBUG: Task 474 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 475 has memoization hash 0ff76c3ae87bee64a6eca41e43bd8917 parsl.dataflow.memoization INFO: Task 475 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfdf80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 475 try 0 launched on executor multi with executor id 344 parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/24ac9036-f36a-4161-9add-f8bf82a2c2d5_characterizeImage_1125543_60.stdout parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/24ac9036-f36a-4161-9add-f8bf82a2c2d5_characterizeImage_1125543_60.stderr parsl.dataflow.dflow INFO: Standard output for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/7c8b3ef6-6543-4e1d-a3ab-da54cca8adb1_isr_1125543_60.stdout parsl.dataflow.dflow INFO: Standard error for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/7c8b3ef6-6543-4e1d-a3ab-da54cca8adb1_isr_1125543_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:09 proc_lsst.multi:146 [INFO] found job 16889838 in provider astro proc_lsst.multi INFO: found job 16889838 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 447 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc9bd110c2ef6ec158ef60d154955597 with result from task 447 parsl.dataflow.dflow DEBUG: Task 447 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 448 has memoization hash 9526a5530710980671cb15d7070814d0 parsl.dataflow.memoization INFO: Task 448 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 448 try 0 launched on executor multi with executor id 345 parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/12fba411-39ec-49d6-bd9a-1f1a444c81fa_characterizeImage_1125542_24.stdout parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/12fba411-39ec-49d6-bd9a-1f1a444c81fa_characterizeImage_1125542_24.stderr parsl.dataflow.dflow INFO: Standard output for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/8b7e9faf-c6b6-48f8-ac57-a8f0c1971e92_isr_1125542_24.stdout parsl.dataflow.dflow INFO: Standard error for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/8b7e9faf-c6b6-48f8-ac57-a8f0c1971e92_isr_1125542_24.stderr parsl.dataflow.dflow INFO: Task 480 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24bd14f767ac2ad56894336e6ec6dddf with result from task 480 parsl.dataflow.dflow DEBUG: Task 480 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 481 has memoization hash 048d9cd72aa2f5216a7c93b0c4e8c647 parsl.dataflow.memoization INFO: Task 481 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfc4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 481 try 0 launched on executor multi with executor id 346 parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/22d9dfc3-d37c-48e0-9c6e-781bf5b939ad_characterizeImage_1125541_58.stdout parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/22d9dfc3-d37c-48e0-9c6e-781bf5b939ad_characterizeImage_1125541_58.stderr parsl.dataflow.dflow INFO: Standard output for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/dbebfd88-a6fe-4ad7-a3d8-50cd03fcd7d0_isr_1125541_58.stdout parsl.dataflow.dflow INFO: Standard error for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/dbebfd88-a6fe-4ad7-a3d8-50cd03fcd7d0_isr_1125541_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:10 proc_lsst.multi:146 [INFO] found job 16889839 in provider astro proc_lsst.multi INFO: found job 16889839 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 468 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87fa9002967079a79f292dc7ff02fe1b with result from task 468 parsl.dataflow.dflow DEBUG: Task 468 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 469 has memoization hash 9ce9806bf52116f7200be01b25914a06 parsl.dataflow.memoization INFO: Task 469 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbfba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 469 try 0 launched on executor multi with executor id 347 parsl.dataflow.dflow INFO: Standard output for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/b9901117-6bed-4880-b228-7e75e54bd27f_characterizeImage_1125541_25.stdout parsl.dataflow.dflow INFO: Standard error for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/b9901117-6bed-4880-b228-7e75e54bd27f_characterizeImage_1125541_25.stderr parsl.dataflow.dflow INFO: Standard output for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/68d45cfc-3e63-49fa-94d4-b12f3209eec9_isr_1125541_25.stdout parsl.dataflow.dflow INFO: Standard error for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/68d45cfc-3e63-49fa-94d4-b12f3209eec9_isr_1125541_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:10 proc_lsst.multi:146 [INFO] found job 16889840 in provider astro proc_lsst.multi INFO: found job 16889840 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 489 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7187be8841c48db85d15bbbdbf70d932 with result from task 489 parsl.dataflow.dflow DEBUG: Task 489 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 490 has memoization hash d4fc8826e0a66c157fa587f5971ae1f2 parsl.dataflow.memoization INFO: Task 490 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfcf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 490 try 0 launched on executor multi with executor id 348 parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/9398b1f1-6694-4d31-a65b-69a4ac8ed3fe_characterizeImage_1125542_14.stdout parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/9398b1f1-6694-4d31-a65b-69a4ac8ed3fe_characterizeImage_1125542_14.stderr parsl.dataflow.dflow INFO: Standard output for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/7d060252-c16a-44d3-a893-2eb73c34682b_isr_1125542_14.stdout parsl.dataflow.dflow INFO: Standard error for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/7d060252-c16a-44d3-a893-2eb73c34682b_isr_1125542_14.stderr parsl.dataflow.dflow INFO: Task 459 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b78f1a5c5e126589bc89fb933dff6664 with result from task 459 parsl.dataflow.dflow DEBUG: Task 459 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 460 has memoization hash 185eda72efae130db3545dc5d94bef0c parsl.dataflow.memoization INFO: Task 460 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfe5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 460 try 0 launched on executor multi with executor id 349 parsl.dataflow.dflow INFO: Standard output for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/ec70f22b-743d-4797-8256-70b4268d8aef_characterizeImage_1125543_47.stdout parsl.dataflow.dflow INFO: Standard error for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/ec70f22b-743d-4797-8256-70b4268d8aef_characterizeImage_1125543_47.stderr parsl.dataflow.dflow INFO: Standard output for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/b9ff6dca-d4b8-405c-b32d-a7add4eb3373_isr_1125543_47.stdout parsl.dataflow.dflow INFO: Standard error for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/b9ff6dca-d4b8-405c-b32d-a7add4eb3373_isr_1125543_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:11 proc_lsst.multi:146 [INFO] found job 16889841 in provider astro proc_lsst.multi INFO: found job 16889841 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 492 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 667efbd298b410847cb3d5be3c40572d with result from task 492 parsl.dataflow.dflow DEBUG: Task 492 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 493 has memoization hash a0ec7f722319fdf3eb3073cf40208f40 parsl.dataflow.memoization INFO: Task 493 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfd260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:11 proc_lsst.multi:146 [INFO] found job 16889842 in provider astro proc_lsst.multi INFO: found job 16889842 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Parsl task 493 try 0 launched on executor multi with executor id 350 parsl.dataflow.dflow INFO: Standard output for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/504cde24-a2ac-4b31-b69e-831aed9871ea_characterizeImage_1125543_40.stdout parsl.dataflow.dflow INFO: Standard error for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/504cde24-a2ac-4b31-b69e-831aed9871ea_characterizeImage_1125543_40.stderr parsl.dataflow.dflow INFO: Standard output for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/6c9363e2-95de-449c-af4a-837cb36ab9e7_isr_1125543_40.stdout parsl.dataflow.dflow INFO: Standard error for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/6c9363e2-95de-449c-af4a-837cb36ab9e7_isr_1125543_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:12 proc_lsst.multi:146 [INFO] found job 16889843 in provider astro proc_lsst.multi INFO: found job 16889843 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:12 proc_lsst.multi:146 [INFO] found job 16889844 in provider astro proc_lsst.multi INFO: found job 16889844 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:13 proc_lsst.multi:146 [INFO] found job 16889845 in provider astro proc_lsst.multi INFO: found job 16889845 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 495 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee424a2c83c11d9100f941aff216546e with result from task 495 parsl.dataflow.dflow DEBUG: Task 495 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 496 has memoization hash 4da8cb6ca8381f7254a105f364174d25 parsl.dataflow.memoization INFO: Task 496 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfe020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 496 try 0 launched on executor multi with executor id 351 parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/fd430661-7ecc-41f5-bc0c-71394e6cb18d_characterizeImage_1125541_38.stdout parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/fd430661-7ecc-41f5-bc0c-71394e6cb18d_characterizeImage_1125541_38.stderr parsl.dataflow.dflow INFO: Standard output for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/718af084-6fcc-44b3-990e-89169eec5142_isr_1125541_38.stdout parsl.dataflow.dflow INFO: Standard error for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/718af084-6fcc-44b3-990e-89169eec5142_isr_1125541_38.stderr parsl.dataflow.dflow INFO: Task 486 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9c353431f9457a4eab7059c934026c1 with result from task 486 parsl.dataflow.dflow DEBUG: Task 486 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 487 has memoization hash 96677fbce79003546f30bab9145e72c0 parsl.dataflow.memoization INFO: Task 487 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfcae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 487 try 0 launched on executor multi with executor id 352 parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/bc4cc82c-0175-43e9-828b-370efa7736e1_characterizeImage_1125542_44.stdout parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/bc4cc82c-0175-43e9-828b-370efa7736e1_characterizeImage_1125542_44.stderr parsl.dataflow.dflow INFO: Standard output for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/4f1707df-9900-4f2e-adb8-d07b2aa0ed4f_isr_1125542_44.stdout parsl.dataflow.dflow INFO: Standard error for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/4f1707df-9900-4f2e-adb8-d07b2aa0ed4f_isr_1125542_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:14 proc_lsst.multi:146 [INFO] found job 16889846 in provider astro proc_lsst.multi INFO: found job 16889846 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:14 proc_lsst.multi:146 [INFO] found job 16889847 in provider astro proc_lsst.multi INFO: found job 16889847 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 498 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc11ddb4c99ef3b9aa88104b6223c1d9 with result from task 498 parsl.dataflow.dflow DEBUG: Task 498 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 499 has memoization hash a5887bd43c3964853c1014dd693b98bb parsl.dataflow.memoization INFO: Task 499 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfdc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 499 try 0 launched on executor multi with executor id 353 parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/e9012e41-2ee4-4e39-88ae-fc47786c5922_characterizeImage_1125542_27.stdout parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/e9012e41-2ee4-4e39-88ae-fc47786c5922_characterizeImage_1125542_27.stderr parsl.dataflow.dflow INFO: Standard output for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/727915c6-5f7c-48da-9c28-9ede6575380e_isr_1125542_27.stdout parsl.dataflow.dflow INFO: Standard error for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/727915c6-5f7c-48da-9c28-9ede6575380e_isr_1125542_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:15 proc_lsst.multi:146 [INFO] found job 16889848 in provider astro proc_lsst.multi INFO: found job 16889848 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 28 try 0 failed parsl.dataflow.dflow INFO: Task 28 marked for retry parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/1fa5e94f-b9b3-4e9f-8222-a315fa75f121_characterizeImage_1125541_2.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/1fa5e94f-b9b3-4e9f-8222-a315fa75f121_characterizeImage_1125541_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash 4ba87d1e2e9a26e3da9bbfc1f5d0ed84 parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149813493740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 1 launched on executor multi with executor id 354 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/1fa5e94f-b9b3-4e9f-8222-a315fa75f121_characterizeImage_1125541_2.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/1fa5e94f-b9b3-4e9f-8222-a315fa75f121_characterizeImage_1125541_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:16 proc_lsst.multi:146 [INFO] found job 16889852 in provider astro proc_lsst.multi INFO: found job 16889852 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:17 proc_lsst.multi:146 [INFO] found job 16889853 in provider astro proc_lsst.multi INFO: found job 16889853 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 543 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2345c4d670ae91a04fbd9cd99da2d2e6 with result from task 543 parsl.dataflow.dflow DEBUG: Task 543 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 544 has memoization hash b105d92feacb79496075f47c0547aafe parsl.dataflow.memoization INFO: Task 544 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811e54b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 544 try 0 launched on executor multi with executor id 355 parsl.dataflow.dflow INFO: Standard output for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/5a1b7098-e4c5-48d1-a753-70a565811ac7_characterizeImage_1125541_50.stdout parsl.dataflow.dflow INFO: Standard error for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/5a1b7098-e4c5-48d1-a753-70a565811ac7_characterizeImage_1125541_50.stderr parsl.dataflow.dflow INFO: Standard output for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/b7887d93-cbd3-4481-b395-1b03449fd3c2_isr_1125541_50.stdout parsl.dataflow.dflow INFO: Standard error for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/b7887d93-cbd3-4481-b395-1b03449fd3c2_isr_1125541_50.stderr parsl.dataflow.dflow INFO: Task 537 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7eb9d402367a0b9db95c33e1919df3e6 with result from task 537 parsl.dataflow.dflow DEBUG: Task 537 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 538 has memoization hash 09d36cf939d6769b0040f1daa01e6e0d parsl.dataflow.memoization INFO: Task 538 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811e54a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 538 try 0 launched on executor multi with executor id 356 parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/6c5df536-70a6-4fdf-820a-6abf12f53449_characterizeImage_1125542_33.stdout parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/6c5df536-70a6-4fdf-820a-6abf12f53449_characterizeImage_1125542_33.stderr parsl.dataflow.dflow INFO: Standard output for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/27fe204f-5cea-414a-a4df-c5c049a02d2d_isr_1125542_33.stdout parsl.dataflow.dflow INFO: Standard error for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/27fe204f-5cea-414a-a4df-c5c049a02d2d_isr_1125542_33.stderr parsl.dataflow.dflow INFO: Task 546 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46fb10abed5c44b8dc31a1a8dfccae21 with result from task 546 parsl.dataflow.dflow DEBUG: Task 546 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 547 has memoization hash 7361d2a176c30636e6f86659cb65109b parsl.dataflow.memoization INFO: Task 547 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811e551c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 547 try 0 launched on executor multi with executor id 357 parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/23e11865-02d8-4f48-afe4-7d613da8a790_characterizeImage_1125541_11.stdout parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/23e11865-02d8-4f48-afe4-7d613da8a790_characterizeImage_1125541_11.stderr parsl.dataflow.dflow INFO: Standard output for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/792d84ae-9dc1-4a77-9f46-27ffe95723fb_isr_1125541_11.stdout parsl.dataflow.dflow INFO: Standard error for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/792d84ae-9dc1-4a77-9f46-27ffe95723fb_isr_1125541_11.stderr parsl.dataflow.dflow INFO: Task 516 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8876e772223f24172c087e6c1d708207 with result from task 516 parsl.dataflow.dflow DEBUG: Task 516 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 517 has memoization hash b882c82f336c8447811a3c97d8229e83 parsl.dataflow.memoization INFO: Task 517 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dff420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 517 try 0 launched on executor multi with executor id 358 parsl.dataflow.dflow INFO: Standard output for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/51730513-f65f-448f-8804-f6e79d3cb228_characterizeImage_1125541_13.stdout parsl.dataflow.dflow INFO: Standard error for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/51730513-f65f-448f-8804-f6e79d3cb228_characterizeImage_1125541_13.stderr parsl.dataflow.dflow INFO: Standard output for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/b714af61-e8b1-403c-8f44-b5f132718532_isr_1125541_13.stdout parsl.dataflow.dflow INFO: Standard error for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/b714af61-e8b1-403c-8f44-b5f132718532_isr_1125541_13.stderr parsl.dataflow.dflow INFO: Task 528 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a64062dac82aba8b0fa5aa0269f33757 with result from task 528 parsl.dataflow.dflow DEBUG: Task 528 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 529 has memoization hash de5a0c7c9879b337ca87238b7355053f parsl.dataflow.memoization INFO: Task 529 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dffec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 529 try 0 launched on executor multi with executor id 359 parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/f38117a3-bf39-451a-b55d-4c3679815476_characterizeImage_1125542_22.stdout parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/f38117a3-bf39-451a-b55d-4c3679815476_characterizeImage_1125542_22.stderr parsl.dataflow.dflow INFO: Standard output for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/10ea726b-4e18-4b91-bb9f-1a4f3383fae5_isr_1125542_22.stdout parsl.dataflow.dflow INFO: Standard error for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/10ea726b-4e18-4b91-bb9f-1a4f3383fae5_isr_1125542_22.stderr parsl.dataflow.dflow INFO: Task 501 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 122023fc7279e3c11d54a38e252d7af9 with result from task 501 parsl.dataflow.dflow DEBUG: Task 501 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 502 has memoization hash 6e52d39c8299a297d686bde9339e6ed9 parsl.dataflow.memoization INFO: Task 502 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfd940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 502 try 0 launched on executor multi with executor id 360 parsl.dataflow.dflow INFO: Standard output for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/a3826850-bb6b-4bb9-8d90-cd7eee67f99c_characterizeImage_1125541_8.stdout parsl.dataflow.dflow INFO: Standard error for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/a3826850-bb6b-4bb9-8d90-cd7eee67f99c_characterizeImage_1125541_8.stderr parsl.dataflow.dflow INFO: Standard output for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/aded3a7c-9bdc-4ddd-9847-fd47ae2757ed_isr_1125541_8.stdout parsl.dataflow.dflow INFO: Standard error for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/aded3a7c-9bdc-4ddd-9847-fd47ae2757ed_isr_1125541_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:20 proc_lsst.multi:146 [INFO] found job 16889854 in provider astro proc_lsst.multi INFO: found job 16889854 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 534 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f6b35b8769947f7da6368709b57f598 with result from task 534 parsl.dataflow.dflow DEBUG: Task 534 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 535 has memoization hash 4c25dceda8cd144d6bf4689440e633da parsl.dataflow.memoization INFO: Task 535 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811e54720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 535 try 0 launched on executor multi with executor id 361 parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/0d5f1499-5ad9-4ece-86c8-c546cf8259ef_characterizeImage_1125543_25.stdout parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/0d5f1499-5ad9-4ece-86c8-c546cf8259ef_characterizeImage_1125543_25.stderr parsl.dataflow.dflow INFO: Standard output for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/bb14c084-19ed-42a1-bdef-e556f3b09a4f_isr_1125543_25.stdout parsl.dataflow.dflow INFO: Standard error for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/bb14c084-19ed-42a1-bdef-e556f3b09a4f_isr_1125543_25.stderr parsl.dataflow.dflow INFO: Task 507 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1dbecb4ff80396b16f40d6d036c71d6a with result from task 507 parsl.dataflow.dflow DEBUG: Task 507 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 508 has memoization hash 1c709ff72f89e1e0550524bd8d378f4a parsl.dataflow.memoization INFO: Task 508 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfeac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 508 try 0 launched on executor multi with executor id 362 parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/69e93b2a-6c40-4982-8ddc-95550993a199_characterizeImage_1125542_46.stdout parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/69e93b2a-6c40-4982-8ddc-95550993a199_characterizeImage_1125542_46.stderr parsl.dataflow.dflow INFO: Standard output for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/b08ae198-cc4e-448e-a00a-ff91b339fa78_isr_1125542_46.stdout parsl.dataflow.dflow INFO: Standard error for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/b08ae198-cc4e-448e-a00a-ff91b339fa78_isr_1125542_46.stderr parsl.dataflow.dflow INFO: Task 519 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0ba54dd01f671fa468ce085ce1bf1d5 with result from task 519 parsl.dataflow.dflow DEBUG: Task 519 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 520 has memoization hash baaa763f25a6830e71de5396b2ef7ed7 parsl.dataflow.memoization INFO: Task 520 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dff740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 520 try 0 launched on executor multi with executor id 363 parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/b33eae3d-2bfb-4ecf-80c5-d5ce9c8e2efa_characterizeImage_1125542_21.stdout parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/b33eae3d-2bfb-4ecf-80c5-d5ce9c8e2efa_characterizeImage_1125542_21.stderr parsl.dataflow.dflow INFO: Standard output for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/f574e8a1-b4f4-4b31-8ef2-5ad1926ffd99_isr_1125542_21.stdout parsl.dataflow.dflow INFO: Standard error for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/f574e8a1-b4f4-4b31-8ef2-5ad1926ffd99_isr_1125542_21.stderr parsl.dataflow.dflow INFO: Task 504 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 843f3157545da7bac893b75742af4ddd with result from task 504 parsl.dataflow.dflow DEBUG: Task 504 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 505 has memoization hash d1e650292e1c58fa620202df3660fba0 parsl.dataflow.memoization INFO: Task 505 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfd620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 505 try 0 launched on executor multi with executor id 364 parsl.dataflow.dflow INFO: Standard output for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/c58b27c7-64ff-409b-8740-6d80faded686_characterizeImage_1125542_26.stdout parsl.dataflow.dflow INFO: Standard error for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/c58b27c7-64ff-409b-8740-6d80faded686_characterizeImage_1125542_26.stderr parsl.dataflow.dflow INFO: Standard output for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/3e2ef4f7-fa8a-4530-ac50-5af6e6cb9d7c_isr_1125542_26.stdout parsl.dataflow.dflow INFO: Standard error for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/3e2ef4f7-fa8a-4530-ac50-5af6e6cb9d7c_isr_1125542_26.stderr parsl.dataflow.dflow INFO: Task 540 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ef5a4d6159f6ee2940a99efbaae0780 with result from task 540 parsl.dataflow.dflow DEBUG: Task 540 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 541 has memoization hash 4a602cf2779dbb4be7d94c1191257a89 parsl.dataflow.memoization INFO: Task 541 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 541 try 0 launched on executor multi with executor id 365 parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/ea7a1c38-9e01-4085-99ed-2ee303a16fc9_characterizeImage_1125542_56.stdout parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/ea7a1c38-9e01-4085-99ed-2ee303a16fc9_characterizeImage_1125542_56.stderr parsl.dataflow.dflow INFO: Standard output for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/dc7f9abe-6435-4546-beb2-def57fda6527_isr_1125542_56.stdout parsl.dataflow.dflow INFO: Standard error for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/dc7f9abe-6435-4546-beb2-def57fda6527_isr_1125542_56.stderr parsl.dataflow.dflow INFO: Task 531 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 365f22383fb935e3453a30cfb5bdea46 with result from task 531 parsl.dataflow.dflow DEBUG: Task 531 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 532 has memoization hash aaf1be6b87663ac5a46a89f49657ccfa parsl.dataflow.memoization INFO: Task 532 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811e54400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 532 try 0 launched on executor multi with executor id 366 parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/668a6fd8-81df-41a0-937e-f6bd18bbe983_characterizeImage_1125543_16.stdout parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/668a6fd8-81df-41a0-937e-f6bd18bbe983_characterizeImage_1125543_16.stderr parsl.dataflow.dflow INFO: Standard output for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/c92d9f01-cc7c-4aa0-8fbe-af9f25fcc24b_isr_1125543_16.stdout parsl.dataflow.dflow INFO: Standard error for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/c92d9f01-cc7c-4aa0-8fbe-af9f25fcc24b_isr_1125543_16.stderr parsl.dataflow.dflow INFO: Task 525 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d91488ca9f40bfab26972ac44d684ac with result from task 525 parsl.dataflow.dflow DEBUG: Task 525 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 526 has memoization hash de3e696d23043e3c87118c283531c444 parsl.dataflow.memoization INFO: Task 526 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dffd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 526 try 0 launched on executor multi with executor id 367 parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/ac4839a9-00a2-4b39-892d-1eb1c9701bef_characterizeImage_1125541_5.stdout parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/ac4839a9-00a2-4b39-892d-1eb1c9701bef_characterizeImage_1125541_5.stderr parsl.dataflow.dflow INFO: Standard output for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/74f4f115-8b6e-4d9f-ae1a-d18d32c582e2_isr_1125541_5.stdout parsl.dataflow.dflow INFO: Standard error for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/74f4f115-8b6e-4d9f-ae1a-d18d32c582e2_isr_1125541_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:20 proc_lsst.multi:146 [INFO] found job 16889855 in provider astro proc_lsst.multi INFO: found job 16889855 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 510 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb9fa70a7e7340f8ef44796c4190e5a7 with result from task 510 parsl.dataflow.dflow DEBUG: Task 510 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 511 has memoization hash aeaef5c7a62a93eb4172c41b6ec62d7a parsl.dataflow.memoization INFO: Task 511 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 511 try 0 launched on executor multi with executor id 368 parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/d17c9d41-1c03-4f32-82ad-9daa739510f0_characterizeImage_1125543_44.stdout parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/d17c9d41-1c03-4f32-82ad-9daa739510f0_characterizeImage_1125543_44.stderr parsl.dataflow.dflow INFO: Standard output for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/dfebb712-057c-4028-9de7-a7cb69bfadc0_isr_1125543_44.stdout parsl.dataflow.dflow INFO: Standard error for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/dfebb712-057c-4028-9de7-a7cb69bfadc0_isr_1125543_44.stderr parsl.dataflow.dflow INFO: Task 513 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42d72c81e261d9519883f07c34ab99bc with result from task 513 parsl.dataflow.dflow DEBUG: Task 513 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 514 has memoization hash c37b72b5a7773da0d49a66c60c9f5922 parsl.dataflow.memoization INFO: Task 514 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dff100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 514 try 0 launched on executor multi with executor id 369 parsl.dataflow.dflow INFO: Standard output for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/50b3a5b4-3d90-43ad-be10-9d269c2929af_characterizeImage_1125542_54.stdout parsl.dataflow.dflow INFO: Standard error for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/50b3a5b4-3d90-43ad-be10-9d269c2929af_characterizeImage_1125542_54.stderr parsl.dataflow.dflow INFO: Standard output for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/d4b9a0aa-1b82-49b0-950e-87a0719a7186_isr_1125542_54.stdout parsl.dataflow.dflow INFO: Standard error for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125542/d4b9a0aa-1b82-49b0-950e-87a0719a7186_isr_1125542_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:21 proc_lsst.multi:146 [INFO] found job 16889856 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889856 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:22 proc_lsst.multi:146 [INFO] found job 16889857 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889857 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.dataflow.dflow INFO: Task 549 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5eb0d6556931082fc9e2a57c991e8012 with result from task 549 parsl.dataflow.dflow DEBUG: Task 549 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 550 has memoization hash 48019b1923fb729bbff3b01f5180158f parsl.dataflow.memoization INFO: Task 550 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811e554e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 550 try 0 launched on executor multi with executor id 370 parsl.dataflow.dflow INFO: Standard output for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/f1a82298-bf28-4111-a38a-67f05dbf5b73_characterizeImage_1125541_56.stdout parsl.dataflow.dflow INFO: Standard error for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/f1a82298-bf28-4111-a38a-67f05dbf5b73_characterizeImage_1125541_56.stderr parsl.dataflow.dflow INFO: Standard output for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/b5d4d438-e9c4-4bdc-8cd8-8cdf13dda8da_isr_1125541_56.stdout parsl.dataflow.dflow INFO: Standard error for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/b5d4d438-e9c4-4bdc-8cd8-8cdf13dda8da_isr_1125541_56.stderr parsl.dataflow.dflow INFO: Task 555 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5fe2571449bdc0abd843d3e3d3f00b7 with result from task 555 parsl.dataflow.dflow DEBUG: Task 555 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 556 has memoization hash 6a48ee9c60beb1a97d2355e3d4a2802d parsl.dataflow.memoization INFO: Task 556 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811e55b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 556 try 0 launched on executor multi with executor id 371 parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/ae187e77-39e8-4e18-be49-3d75fc85361c_characterizeImage_1125543_52.stdout parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/ae187e77-39e8-4e18-be49-3d75fc85361c_characterizeImage_1125543_52.stderr parsl.dataflow.dflow INFO: Standard output for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/e4d82c4f-3770-4a83-98e1-f7e65c4643a8_isr_1125543_52.stdout parsl.dataflow.dflow INFO: Standard error for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125543/e4d82c4f-3770-4a83-98e1-f7e65c4643a8_isr_1125543_52.stderr parsl.dataflow.dflow INFO: Task 552 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd855c701822a217a3a240619599f977 with result from task 552 parsl.dataflow.dflow DEBUG: Task 552 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 553 has memoization hash c77bf4ecacf7555783b02f2ee417cd17 parsl.dataflow.memoization INFO: Task 553 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811e55800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 553 try 0 launched on executor multi with executor id 372 parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/e698a0fa-658c-4a4c-8ef1-d67979152518_characterizeImage_1125541_7.stdout parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/e698a0fa-658c-4a4c-8ef1-d67979152518_characterizeImage_1125541_7.stderr parsl.dataflow.dflow INFO: Standard output for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/6d75e0ef-0c9e-4be1-8ab6-94ccaf31b0ce_isr_1125541_7.stdout parsl.dataflow.dflow INFO: Standard error for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/6d75e0ef-0c9e-4be1-8ab6-94ccaf31b0ce_isr_1125541_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:22 proc_lsst.multi:146 [INFO] found job 16889858 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889858 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.dataflow.dflow INFO: Task 522 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3bcca5f836a8253c0c1a7034aa28565b with result from task 522 parsl.dataflow.dflow DEBUG: Task 522 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 523 has memoization hash c1b3c55bf24bd1ef283947cdcb22bbc2 parsl.dataflow.memoization INFO: Task 523 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dffa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 523 try 0 launched on executor multi with executor id 373 parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/cc979036-214f-4d4d-a941-3b7629ea62f5_characterizeImage_1125541_52.stdout parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/cc979036-214f-4d4d-a941-3b7629ea62f5_characterizeImage_1125541_52.stderr parsl.dataflow.dflow INFO: Standard output for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/e18f27f8-3241-4be8-bf69-9c83faac0fe9_isr_1125541_52.stdout parsl.dataflow.dflow INFO: Standard error for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/isr/1125541/e18f27f8-3241-4be8-bf69-9c83faac0fe9_isr_1125541_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:23 proc_lsst.multi:146 [INFO] found job 16889859 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889859 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:23 proc_lsst.multi:146 [INFO] found job 16889860 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889860 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:23 proc_lsst.multi:146 [INFO] found job 16889861 in provider escience_a40 proc_lsst.multi INFO: found job 16889861 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:23 proc_lsst.multi:146 [INFO] found job 16889862 in provider escience_a40 proc_lsst.multi INFO: found job 16889862 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:24 proc_lsst.multi:146 [INFO] found job 16889863 in provider escience_a40 proc_lsst.multi INFO: found job 16889863 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:24 proc_lsst.multi:146 [INFO] found job 16889864 in provider escience_a40 proc_lsst.multi INFO: found job 16889864 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:24 proc_lsst.multi:146 [INFO] found job 16889865 in provider escience_a40 proc_lsst.multi INFO: found job 16889865 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:24 proc_lsst.multi:146 [INFO] found job 16889866 in provider escience_a40 proc_lsst.multi INFO: found job 16889866 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:24 proc_lsst.multi:146 [INFO] found job 16889867 in provider escience_a40 proc_lsst.multi INFO: found job 16889867 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:24 proc_lsst.multi:146 [INFO] found job 16889868 in provider escience_a40 proc_lsst.multi INFO: found job 16889868 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:25 proc_lsst.multi:146 [INFO] found job 16889869 in provider escience_a40 proc_lsst.multi INFO: found job 16889869 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:25 proc_lsst.multi:146 [INFO] found job 16889870 in provider escience_a40 proc_lsst.multi INFO: found job 16889870 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:25 proc_lsst.multi:146 [INFO] found job 16889871 in provider escience_a40 proc_lsst.multi INFO: found job 16889871 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22643663051856 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22643663051856 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 22 try 0 failed parsl.dataflow.dflow INFO: Task 22 marked for retry parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/2000f40d-8eda-44b7-bbea-08ecd9dbb933_characterizeImage_1125543_2.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/2000f40d-8eda-44b7-bbea-08ecd9dbb933_characterizeImage_1125543_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash 70ca8e34191ccafee926a0ba43c4500c parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149813493d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 1 launched on executor multi with executor id 374 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/2000f40d-8eda-44b7-bbea-08ecd9dbb933_characterizeImage_1125543_2.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/2000f40d-8eda-44b7-bbea-08ecd9dbb933_characterizeImage_1125543_2.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22643663051856 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 121 try 0 failed parsl.dataflow.dflow INFO: Task 121 marked for retry parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/4e8791e5-4edd-4a9d-ac14-e1b97579fc05_characterizeImage_1125542_2.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/4e8791e5-4edd-4a9d-ac14-e1b97579fc05_characterizeImage_1125542_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 121 has memoization hash bb50db9f13537bf93733584b053700e1 parsl.dataflow.memoization INFO: Task 121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ead260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 121 try 1 launched on executor multi with executor id 375 parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/4e8791e5-4edd-4a9d-ac14-e1b97579fc05_characterizeImage_1125542_2.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/4e8791e5-4edd-4a9d-ac14-e1b97579fc05_characterizeImage_1125542_2.stderr 2024-03-07 10:24:35 proc_lsst.multi:146 [INFO] found job 24403 in provider local proc_lsst.multi INFO: found job 24403 in provider local 2024-03-07 10:24:35 proc_lsst.multi:146 [INFO] found job 16889823 in provider astro proc_lsst.multi INFO: found job 16889823 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:36 proc_lsst.multi:146 [INFO] found job 16889824 in provider astro proc_lsst.multi INFO: found job 16889824 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:37 proc_lsst.multi:146 [INFO] found job 16889825 in provider astro proc_lsst.multi INFO: found job 16889825 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:38 proc_lsst.multi:146 [INFO] found job 16889826 in provider astro proc_lsst.multi INFO: found job 16889826 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:39 proc_lsst.multi:146 [INFO] found job 16889827 in provider astro proc_lsst.multi INFO: found job 16889827 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:40 proc_lsst.multi:146 [INFO] found job 16889828 in provider astro proc_lsst.multi INFO: found job 16889828 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:41 proc_lsst.multi:146 [INFO] found job 16889829 in provider astro proc_lsst.multi INFO: found job 16889829 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:42 proc_lsst.multi:146 [INFO] found job 16889830 in provider astro proc_lsst.multi INFO: found job 16889830 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:43 proc_lsst.multi:146 [INFO] found job 16889831 in provider astro proc_lsst.multi INFO: found job 16889831 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:44 proc_lsst.multi:146 [INFO] found job 16889832 in provider astro proc_lsst.multi INFO: found job 16889832 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:45 proc_lsst.multi:146 [INFO] found job 16889833 in provider astro proc_lsst.multi INFO: found job 16889833 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:46 proc_lsst.multi:146 [INFO] found job 16889834 in provider astro proc_lsst.multi INFO: found job 16889834 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:47 proc_lsst.multi:146 [INFO] found job 16889835 in provider astro proc_lsst.multi INFO: found job 16889835 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:49 proc_lsst.multi:146 [INFO] found job 16889836 in provider astro proc_lsst.multi INFO: found job 16889836 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:50 proc_lsst.multi:146 [INFO] found job 16889837 in provider astro proc_lsst.multi INFO: found job 16889837 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:51 proc_lsst.multi:146 [INFO] found job 16889838 in provider astro proc_lsst.multi INFO: found job 16889838 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:52 proc_lsst.multi:146 [INFO] found job 16889839 in provider astro proc_lsst.multi INFO: found job 16889839 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:53 proc_lsst.multi:146 [INFO] found job 16889840 in provider astro proc_lsst.multi INFO: found job 16889840 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:54 proc_lsst.multi:146 [INFO] found job 16889841 in provider astro proc_lsst.multi INFO: found job 16889841 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:55 proc_lsst.multi:146 [INFO] found job 16889842 in provider astro proc_lsst.multi INFO: found job 16889842 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a7ae93bc72d2a6ee3e81f000095805f with result from task 10 parsl.dataflow.dflow DEBUG: Task 10 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 11 has memoization hash 4baf95301d3815441b7425351838dc79 parsl.dataflow.memoization INFO: Task 11 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134919e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 376 parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/0e5bf43c-a446-4ee3-806a-402d7664fbfd_calibrate_1125541_29.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/0e5bf43c-a446-4ee3-806a-402d7664fbfd_calibrate_1125541_29.stderr parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/e8e108f9-68f6-4453-b369-9264be7a7909_characterizeImage_1125541_29.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/e8e108f9-68f6-4453-b369-9264be7a7909_characterizeImage_1125541_29.stderr parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7090026edd87cf9b97a0e5d2646e7144 with result from task 43 parsl.dataflow.dflow DEBUG: Task 43 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 44 has memoization hash 7ce16498d71a0713f69072c2aa289f81 parsl.dataflow.memoization INFO: Task 44 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149813492c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 377 parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/e7ed72a0-5d2f-4843-ae78-8bceaaea2489_calibrate_1125543_8.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/e7ed72a0-5d2f-4843-ae78-8bceaaea2489_calibrate_1125543_8.stderr parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/e7e70b68-1337-4398-84f4-66273376df39_characterizeImage_1125543_8.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/e7e70b68-1337-4398-84f4-66273376df39_characterizeImage_1125543_8.stderr parsl.dataflow.dflow INFO: Task 25 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1ae75f06d5fb540eb016e6ae42e1767 with result from task 25 parsl.dataflow.dflow DEBUG: Task 25 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 26 has memoization hash ea497160b6500b6fbff6c8ec12c90eeb parsl.dataflow.memoization INFO: Task 26 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149813493ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 378 parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/be26deb9-1cca-432b-822a-6a6f41138fa2_calibrate_1125541_26.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/be26deb9-1cca-432b-822a-6a6f41138fa2_calibrate_1125541_26.stderr parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/6d8804b9-fd0d-453c-8cc2-8690c4925166_characterizeImage_1125541_26.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/6d8804b9-fd0d-453c-8cc2-8690c4925166_characterizeImage_1125541_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:56 proc_lsst.multi:146 [INFO] found job 16889843 in provider astro proc_lsst.multi INFO: found job 16889843 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:57 proc_lsst.multi:146 [INFO] found job 16889844 in provider astro proc_lsst.multi INFO: found job 16889844 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:58 proc_lsst.multi:146 [INFO] found job 16889845 in provider astro proc_lsst.multi INFO: found job 16889845 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:24:59 proc_lsst.multi:146 [INFO] found job 16889846 in provider astro proc_lsst.multi INFO: found job 16889846 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:00 proc_lsst.multi:146 [INFO] found job 16889847 in provider astro proc_lsst.multi INFO: found job 16889847 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:01 proc_lsst.multi:146 [INFO] found job 16889848 in provider astro proc_lsst.multi INFO: found job 16889848 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:02 proc_lsst.multi:146 [INFO] found job 16889852 in provider astro proc_lsst.multi INFO: found job 16889852 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:03 proc_lsst.multi:146 [INFO] found job 16889853 in provider astro proc_lsst.multi INFO: found job 16889853 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8cccf12a7c4bc74c3bdbf121edb79ca7 with result from task 100 parsl.dataflow.dflow DEBUG: Task 100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 101 has memoization hash a72348daee8ac6e4cd0b201e7761b288 parsl.dataflow.memoization INFO: Task 101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b7a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 101 try 0 launched on executor multi with executor id 379 parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/d376594e-f605-4102-9a56-98b29324a468_calibrate_1125541_6.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/d376594e-f605-4102-9a56-98b29324a468_calibrate_1125541_6.stderr parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/a332ba6f-3076-41fa-8c9f-ddab6f6ef9d6_characterizeImage_1125541_6.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/a332ba6f-3076-41fa-8c9f-ddab6f6ef9d6_characterizeImage_1125541_6.stderr parsl.dataflow.dflow INFO: Task 52 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ade0cc720fef4cd0cfda872368bd952 with result from task 52 parsl.dataflow.dflow DEBUG: Task 52 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 53 has memoization hash 9c9789f14cc8d65ae6792e4dae48cf2d parsl.dataflow.memoization INFO: Task 53 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b4860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 380 parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/adf89e09-fb66-4896-b320-94334b35eeae_calibrate_1125541_4.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/adf89e09-fb66-4896-b320-94334b35eeae_calibrate_1125541_4.stderr parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/b961916b-5350-414f-b0ce-fdd9e2576d31_characterizeImage_1125541_4.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/b961916b-5350-414f-b0ce-fdd9e2576d31_characterizeImage_1125541_4.stderr parsl.dataflow.dflow INFO: Task 55 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42320877e3b4cb007f5285f4e164ae5e with result from task 55 parsl.dataflow.dflow DEBUG: Task 55 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 56 has memoization hash d18b5145681869cdb2f5ade38436e821 parsl.dataflow.memoization INFO: Task 56 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b4b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 381 parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/c559a43c-08ee-4e45-bdcd-1f5e483f1fb5_calibrate_1125541_44.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/c559a43c-08ee-4e45-bdcd-1f5e483f1fb5_calibrate_1125541_44.stderr parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/ffd86f7f-6b4a-4f6c-8980-400bed9e8685_characterizeImage_1125541_44.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/ffd86f7f-6b4a-4f6c-8980-400bed9e8685_characterizeImage_1125541_44.stderr parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56eb355742d97ee42eaf7a6eff97a63a with result from task 112 parsl.dataflow.dflow DEBUG: Task 112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 113 has memoization hash d7dc571e2b8b4dc1876da8e9c7233b8d parsl.dataflow.memoization INFO: Task 113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eac720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 382 parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/410d034b-9d84-45d6-9ea0-d64343b7e5ce_calibrate_1125543_20.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/410d034b-9d84-45d6-9ea0-d64343b7e5ce_calibrate_1125543_20.stderr parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/3c6d60e9-2e23-4182-9a1f-b6c13b27bda3_characterizeImage_1125543_20.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/3c6d60e9-2e23-4182-9a1f-b6c13b27bda3_characterizeImage_1125543_20.stderr parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04cf994ee15a110c1e592fedf7a8841e with result from task 13 parsl.dataflow.dflow DEBUG: Task 13 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 14 has memoization hash efd943edd2990c44034c182934f99a5f parsl.dataflow.memoization INFO: Task 14 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149813491c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 383 parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/763b996f-91a3-4df3-845d-2dd87bff2361_calibrate_1125542_58.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/763b996f-91a3-4df3-845d-2dd87bff2361_calibrate_1125542_58.stderr parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/3cb571e7-d850-4e76-8385-50f414c88b80_characterizeImage_1125542_58.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/3cb571e7-d850-4e76-8385-50f414c88b80_characterizeImage_1125542_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:05 proc_lsst.multi:146 [INFO] found job 16889854 in provider astro proc_lsst.multi INFO: found job 16889854 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:06 proc_lsst.multi:146 [INFO] found job 16889855 in provider astro proc_lsst.multi INFO: found job 16889855 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0314cc2a6fa773ab0f380f7fb6a27109 with result from task 91 parsl.dataflow.dflow DEBUG: Task 91 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash a68c847280705bd81b4606c5cb4bb629 parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b7100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 384 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/5296b656-648d-408c-bbdc-ba228dd0d9c3_calibrate_1125541_28.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/5296b656-648d-408c-bbdc-ba228dd0d9c3_calibrate_1125541_28.stderr parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/3f642c90-d4fb-4572-96d4-1f7e0d1da4ea_characterizeImage_1125541_28.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/3f642c90-d4fb-4572-96d4-1f7e0d1da4ea_characterizeImage_1125541_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:07 proc_lsst.multi:146 [INFO] found job 16889856 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889856 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:09 proc_lsst.multi:146 [INFO] found job 16889857 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889857 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb37ab766fdf6e2272f149b7e85ec6c1 with result from task 7 parsl.dataflow.dflow DEBUG: Task 7 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 8 has memoization hash 1c6b9bcf6f5275e7025673f6c760f0dc parsl.dataflow.memoization INFO: Task 8 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149813490fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 385 parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/739003f2-42dd-4236-a694-27598d980cc4_calibrate_1125542_38.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/739003f2-42dd-4236-a694-27598d980cc4_calibrate_1125542_38.stderr parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/650854cd-95fa-4b60-9fa0-63e43a3d71c3_characterizeImage_1125542_38.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/650854cd-95fa-4b60-9fa0-63e43a3d71c3_characterizeImage_1125542_38.stderr parsl.dataflow.dflow INFO: Task 40 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bdb38e1d7c8156d8c146ac8711c03e14 with result from task 40 parsl.dataflow.dflow DEBUG: Task 40 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 41 has memoization hash 3618212f7652211d22f212017b251224 parsl.dataflow.memoization INFO: Task 41 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134928e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 386 parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/85404648-4db4-4acd-9d37-7bfeea39527c_calibrate_1125541_59.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/85404648-4db4-4acd-9d37-7bfeea39527c_calibrate_1125541_59.stderr parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/ef7a984d-1716-47ce-8f2a-71c7ddf8f425_characterizeImage_1125541_59.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/ef7a984d-1716-47ce-8f2a-71c7ddf8f425_characterizeImage_1125541_59.stderr parsl.dataflow.dflow INFO: Task 61 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc75b1776900b1fc32eab7eab92da95d with result from task 61 parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:10 proc_lsst.multi:146 [INFO] found job 16889858 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889858 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.dataflow.dflow DEBUG: Task 61 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 62 has memoization hash 0267a09c85bc8f5e597ab8b7c98325f0 parsl.dataflow.memoization INFO: Task 62 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b51c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 387 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/0402a9f9-2a34-4b02-8bb0-b1437ee8126b_calibrate_1125541_27.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/0402a9f9-2a34-4b02-8bb0-b1437ee8126b_calibrate_1125541_27.stderr parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/cc1be6fe-47aa-44c6-a99c-202d1fa14d96_characterizeImage_1125541_27.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/cc1be6fe-47aa-44c6-a99c-202d1fa14d96_characterizeImage_1125541_27.stderr parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f79ae5a46944ffa0795965554d0378c9 with result from task 16 parsl.dataflow.dflow DEBUG: Task 16 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 17 has memoization hash 4e2fe06ebf6d6afd19473ee017f56023 parsl.dataflow.memoization INFO: Task 17 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134922a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 388 parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/bedbcfe1-3275-4d33-b0d3-d60632191ee9_calibrate_1125542_60.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/bedbcfe1-3275-4d33-b0d3-d60632191ee9_calibrate_1125542_60.stderr parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/09517e27-ffc4-4af9-9b67-580e7029299a_characterizeImage_1125542_60.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/09517e27-ffc4-4af9-9b67-580e7029299a_characterizeImage_1125542_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:11 proc_lsst.multi:146 [INFO] found job 16889859 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889859 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3906a35294d93dc5e50a6d26ceac117f with result from task 94 parsl.dataflow.dflow DEBUG: Task 94 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 95 has memoization hash 15888dd33db9794b59ccee6c0a66a255 parsl.dataflow.memoization INFO: Task 95 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b7420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 389 parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/d43c382d-709b-4c4a-880d-96263b69f342_calibrate_1125541_37.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/d43c382d-709b-4c4a-880d-96263b69f342_calibrate_1125541_37.stderr parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/4c7882d9-cd30-48db-9424-6fc004e162ed_characterizeImage_1125541_37.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/4c7882d9-cd30-48db-9424-6fc004e162ed_characterizeImage_1125541_37.stderr parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba9044c402c1fdf915e06a95ef66d4d6 with result from task 46 parsl.dataflow.dflow DEBUG: Task 46 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 47 has memoization hash 1f528cbad6fd04dfcdaec8f1ed0bf18a parsl.dataflow.memoization INFO: Task 47 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b4220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 390 parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/ba6ae70d-58e1-45f6-ad0b-6512534f3dc6_calibrate_1125542_40.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/ba6ae70d-58e1-45f6-ad0b-6512534f3dc6_calibrate_1125542_40.stderr parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/245218ac-2d70-4462-b7ef-85977223dccf_characterizeImage_1125542_40.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/245218ac-2d70-4462-b7ef-85977223dccf_characterizeImage_1125542_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:12 proc_lsst.multi:146 [INFO] found job 16889860 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889860 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 480108c646892d704a141e9b13b188ad with result from task 88 parsl.dataflow.dflow DEBUG: Task 88 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 89 has memoization hash bf9c988b222399f81109042b506c923e parsl.dataflow.memoization INFO: Task 89 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b6de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 391 parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/188141af-b21d-45b2-87e9-5314b492275e_calibrate_1125541_33.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/188141af-b21d-45b2-87e9-5314b492275e_calibrate_1125541_33.stderr parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/61bb8955-9b9f-4f44-ad6b-048dd09597ac_characterizeImage_1125541_33.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/61bb8955-9b9f-4f44-ad6b-048dd09597ac_characterizeImage_1125541_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:13 proc_lsst.multi:146 [INFO] found job 16889861 in provider escience_a40 proc_lsst.multi INFO: found job 16889861 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:14 proc_lsst.multi:146 [INFO] found job 16889862 in provider escience_a40 proc_lsst.multi INFO: found job 16889862 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 169 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87be9b9c2f404a076f0a3f1f02c50ba3 with result from task 169 parsl.dataflow.dflow DEBUG: Task 169 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 170 has memoization hash 1669faa1d661b0df13ae4af2025466e8 parsl.dataflow.memoization INFO: Task 170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed42c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 170 try 0 launched on executor multi with executor id 392 parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/a76a74c5-2d68-4aed-abcf-2e47f92e1edd_calibrate_1125541_51.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/a76a74c5-2d68-4aed-abcf-2e47f92e1edd_calibrate_1125541_51.stderr parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/36eca3ca-23f9-46e8-81e8-c55fefdd19e6_characterizeImage_1125541_51.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/36eca3ca-23f9-46e8-81e8-c55fefdd19e6_characterizeImage_1125541_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:15 proc_lsst.multi:146 [INFO] found job 16889863 in provider escience_a40 proc_lsst.multi INFO: found job 16889863 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:16 proc_lsst.multi:146 [INFO] found job 16889864 in provider escience_a40 proc_lsst.multi INFO: found job 16889864 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27f5eb41313e35db05b6988ea66d3934 with result from task 85 parsl.dataflow.dflow DEBUG: Task 85 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 86 has memoization hash aed6556380bf4dec5757f1d521b08431 parsl.dataflow.memoization INFO: Task 86 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b6ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 393 parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/9fc65122-79ab-41e2-9c53-1764893f9833_calibrate_1125543_32.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/9fc65122-79ab-41e2-9c53-1764893f9833_calibrate_1125543_32.stderr parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/1fd8afc2-4144-4bc2-8187-66fb8cbacb69_characterizeImage_1125543_32.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/1fd8afc2-4144-4bc2-8187-66fb8cbacb69_characterizeImage_1125543_32.stderr parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ada5d702006821bb15d823ad89d539ba with result from task 106 parsl.dataflow.dflow DEBUG: Task 106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 107 has memoization hash fc496866428dbf2fc98b8202f5168e96 parsl.dataflow.memoization INFO: Task 107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eac0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 394 parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/a9da3e46-9a24-47bf-80da-dc81b0920ca5_calibrate_1125541_62.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/a9da3e46-9a24-47bf-80da-dc81b0920ca5_calibrate_1125541_62.stderr parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/0616b4dc-39f0-4043-88ca-b4579b300270_characterizeImage_1125541_62.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/0616b4dc-39f0-4043-88ca-b4579b300270_characterizeImage_1125541_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:17 proc_lsst.multi:146 [INFO] found job 16889865 in provider escience_a40 proc_lsst.multi INFO: found job 16889865 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 148 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 259f9ef5b9e5f5b16e74f3bd6e9f300c with result from task 148 parsl.dataflow.dflow DEBUG: Task 148 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 149 has memoization hash c466fed24cec5e5482e52d068fa1adaa parsl.dataflow.memoization INFO: Task 149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eaeca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 149 try 0 launched on executor multi with executor id 395 parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/cf79c1d8-1456-41d8-a784-34fd6545652a_calibrate_1125542_31.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/cf79c1d8-1456-41d8-a784-34fd6545652a_calibrate_1125542_31.stderr parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/556c2692-5113-4bd6-9729-38b4a7b18426_characterizeImage_1125542_31.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/556c2692-5113-4bd6-9729-38b4a7b18426_characterizeImage_1125542_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:18 proc_lsst.multi:146 [INFO] found job 16889866 in provider escience_a40 proc_lsst.multi INFO: found job 16889866 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 49 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7409d0aa756f682a5edbf092cd1ab125 with result from task 49 parsl.dataflow.dflow DEBUG: Task 49 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 50 has memoization hash ee0b17a415a5a4af1b7111e6ea841553 parsl.dataflow.memoization INFO: Task 50 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b4540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 396 parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/ea0297c2-4166-4889-93a0-2934af51af04_calibrate_1125543_26.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/ea0297c2-4166-4889-93a0-2934af51af04_calibrate_1125543_26.stderr parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/f97752e4-31ca-4cad-8584-6e83fb78ebc5_characterizeImage_1125543_26.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/f97752e4-31ca-4cad-8584-6e83fb78ebc5_characterizeImage_1125543_26.stderr parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48eb64fce03e605769968dc6db80f1d3 with result from task 58 parsl.dataflow.dflow DEBUG: Task 58 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 59 has memoization hash 6bf5c63c77bfba96568fb2cd00fcd63d parsl.dataflow.memoization INFO: Task 59 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b4ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 397 parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/69949520-f6f6-4e87-8fa4-02c704e8d595_calibrate_1125541_10.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/69949520-f6f6-4e87-8fa4-02c704e8d595_calibrate_1125541_10.stderr parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/1aa4c0b0-1830-44f8-ad6e-fa52db6e2705_characterizeImage_1125541_10.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/1aa4c0b0-1830-44f8-ad6e-fa52db6e2705_characterizeImage_1125541_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:19 proc_lsst.multi:146 [INFO] found job 16889867 in provider escience_a40 proc_lsst.multi INFO: found job 16889867 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 1 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32b468b441b7b25556e9c43977b6ae81 with result from task 1 parsl.dataflow.dflow DEBUG: Task 1 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2 has memoization hash 66917e4fc20a2757c587932540d955b7 parsl.dataflow.memoization INFO: Task 2 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149813491080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 398 parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/66fe5e1a-93cb-4636-95f2-7243aaab6841_calibrate_1125543_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/66fe5e1a-93cb-4636-95f2-7243aaab6841_calibrate_1125543_62.stderr parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/9f1e39ae-2d1f-4269-87fa-1968fceea5b6_characterizeImage_1125543_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/9f1e39ae-2d1f-4269-87fa-1968fceea5b6_characterizeImage_1125543_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:20 proc_lsst.multi:146 [INFO] found job 16889868 in provider escience_a40 proc_lsst.multi INFO: found job 16889868 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:21 proc_lsst.multi:146 [INFO] found job 16889869 in provider escience_a40 proc_lsst.multi INFO: found job 16889869 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9f127cd16c7a1e2480589857ca9576d with result from task 37 parsl.dataflow.dflow DEBUG: Task 37 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 38 has memoization hash 22087684b7b68cfe8ddbf7081ce0241a parsl.dataflow.memoization INFO: Task 38 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149813492d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 399 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/4be18a14-27a3-4fc5-b1b0-d1376f7a60f3_calibrate_1125543_50.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/4be18a14-27a3-4fc5-b1b0-d1376f7a60f3_calibrate_1125543_50.stderr parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/487dcdd1-07c4-4224-98f0-e15c06e63859_characterizeImage_1125543_50.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/487dcdd1-07c4-4224-98f0-e15c06e63859_characterizeImage_1125543_50.stderr parsl.dataflow.dflow INFO: Task 139 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 383be55477a7e0924ef404c8aa1a7e88 with result from task 139 parsl.dataflow.dflow DEBUG: Task 139 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 140 has memoization hash dfd6144bc06f5dd00a6f3b40ca2a2150 parsl.dataflow.memoization INFO: Task 140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eae340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 400 parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/23a1bacc-1b3c-46da-8122-92bf30499238_calibrate_1125543_9.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/23a1bacc-1b3c-46da-8122-92bf30499238_calibrate_1125543_9.stderr parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/fcbe2b7d-97c2-4b64-8f94-4a3a56772d80_characterizeImage_1125543_9.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/fcbe2b7d-97c2-4b64-8f94-4a3a56772d80_characterizeImage_1125543_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:22 proc_lsst.multi:146 [INFO] found job 16889870 in provider escience_a40 proc_lsst.multi INFO: found job 16889870 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 64 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44403bd3b112d69fed86835d166c6dad with result from task 64 parsl.dataflow.dflow DEBUG: Task 64 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 65 has memoization hash e0e9ca845c7f92bedf2886bb08c21369 parsl.dataflow.memoization INFO: Task 65 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b54e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 401 parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/be495a56-3583-4257-8834-aa242c38745e_calibrate_1125543_38.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/be495a56-3583-4257-8834-aa242c38745e_calibrate_1125543_38.stderr parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/69a3c0c9-15b2-43f6-850e-1fb3967725fe_characterizeImage_1125543_38.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/69a3c0c9-15b2-43f6-850e-1fb3967725fe_characterizeImage_1125543_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:23 proc_lsst.multi:146 [INFO] found job 16889871 in provider escience_a40 proc_lsst.multi INFO: found job 16889871 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22643663051856 2024-03-07 10:25:24 proc_lsst.multi:146 [INFO] found job 24403 in provider local proc_lsst.multi INFO: found job 24403 in provider local 2024-03-07 10:25:24 proc_lsst.multi:146 [INFO] found job 16889823 in provider astro proc_lsst.multi INFO: found job 16889823 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0f0172b734bba78dd9c3442d0f01213 with result from task 19 parsl.dataflow.dflow DEBUG: Task 19 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 20 has memoization hash 92a504ec48269c7e78c506044637db59 parsl.dataflow.memoization INFO: Task 20 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149813492200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 402 parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/94ed5159-e9d5-449a-84c6-198d8f415f8c_calibrate_1125543_12.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/94ed5159-e9d5-449a-84c6-198d8f415f8c_calibrate_1125543_12.stderr parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/0b1aa223-bd70-44ee-97ea-bd51a0f7a410_characterizeImage_1125543_12.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/0b1aa223-bd70-44ee-97ea-bd51a0f7a410_characterizeImage_1125543_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:25 proc_lsst.multi:146 [INFO] found job 16889824 in provider astro proc_lsst.multi INFO: found job 16889824 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 124 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59314a23c89e2216e78d8f3a4fe9b786 with result from task 124 parsl.dataflow.dflow DEBUG: Task 124 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 125 has memoization hash e222da0c3baf4a59ccaa46649b7b6f18 parsl.dataflow.memoization INFO: Task 125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ead3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 125 try 0 launched on executor multi with executor id 403 parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/ec2ef3d7-b3f1-4588-b744-7a163162120d_calibrate_1125541_21.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/ec2ef3d7-b3f1-4588-b744-7a163162120d_calibrate_1125541_21.stderr parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/044c47ec-8ed9-4411-bdda-2eaa74150658_characterizeImage_1125541_21.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/044c47ec-8ed9-4411-bdda-2eaa74150658_characterizeImage_1125541_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:26 proc_lsst.multi:146 [INFO] found job 16889825 in provider astro proc_lsst.multi INFO: found job 16889825 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25202bcc451235f74ef5e0c00afbfd4a with result from task 103 parsl.dataflow.dflow DEBUG: Task 103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 104 has memoization hash 2819f0434648c88dc7d180deb8a11aa3 parsl.dataflow.memoization INFO: Task 104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b7d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 404 parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/d828b9c9-5382-415b-8d6f-35bcb8fd505e_calibrate_1125543_27.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/d828b9c9-5382-415b-8d6f-35bcb8fd505e_calibrate_1125543_27.stderr parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/dcc846d1-02f4-4aca-8c47-747b364632ab_characterizeImage_1125543_27.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/dcc846d1-02f4-4aca-8c47-747b364632ab_characterizeImage_1125543_27.stderr parsl.dataflow.dflow INFO: Task 127 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d540db0ea0144ac993ececb67e9d6753 with result from task 127 parsl.dataflow.dflow DEBUG: Task 127 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 128 has memoization hash 049fe27de73bb79a6268525202de1a77 parsl.dataflow.memoization INFO: Task 128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ead6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 405 parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/3814389b-b750-41cb-91f8-79c8acef780a_calibrate_1125542_43.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/3814389b-b750-41cb-91f8-79c8acef780a_calibrate_1125542_43.stderr parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/2274241f-ebb6-4910-8b78-c93b80436f89_characterizeImage_1125542_43.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/2274241f-ebb6-4910-8b78-c93b80436f89_characterizeImage_1125542_43.stderr parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c931078d39c55770e995989b65c8f4d0 with result from task 31 parsl.dataflow.dflow DEBUG: Task 31 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 32 has memoization hash 1b6db47ea26787cb09ea840294dcefaf parsl.dataflow.memoization INFO: Task 32 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149813493600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 406 parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/a4e62d32-0ab0-4e18-87f2-946360180960_calibrate_1125542_50.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/a4e62d32-0ab0-4e18-87f2-946360180960_calibrate_1125542_50.stderr parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/5dc30535-2070-47fd-ad17-245b576c8b10_characterizeImage_1125542_50.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/5dc30535-2070-47fd-ad17-245b576c8b10_characterizeImage_1125542_50.stderr parsl.dataflow.dflow INFO: Task 97 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 438edc74a6cc3b90056f886c6a6a321c with result from task 97 parsl.dataflow.dflow DEBUG: Task 97 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 98 has memoization hash 79d973e005cb196836076dd99a5efd59 parsl.dataflow.memoization INFO: Task 98 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b7740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 407 parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/ad1d64e1-9f10-4736-b772-fd5954ae0eeb_calibrate_1125543_39.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/ad1d64e1-9f10-4736-b772-fd5954ae0eeb_calibrate_1125543_39.stderr parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/b0494e64-f1f7-4fd4-8c1f-8e87c1eb9fde_characterizeImage_1125543_39.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/b0494e64-f1f7-4fd4-8c1f-8e87c1eb9fde_characterizeImage_1125543_39.stderr parsl.dataflow.dflow INFO: Task 145 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d795b1889db9d7eeb8ee255b1c45c62b with result from task 145 parsl.dataflow.dflow DEBUG: Task 145 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 146 has memoization hash 9267b34e30014cf8c85da849d3a4ff51 parsl.dataflow.memoization INFO: Task 146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eae980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 408 parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/f183129a-5200-4437-bf44-29e5311c6a4c_calibrate_1125541_42.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/f183129a-5200-4437-bf44-29e5311c6a4c_calibrate_1125541_42.stderr parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/34e42774-6699-4a7c-a219-19e62bf778b2_characterizeImage_1125541_42.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/34e42774-6699-4a7c-a219-19e62bf778b2_characterizeImage_1125541_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:27 proc_lsst.multi:146 [INFO] found job 16889826 in provider astro proc_lsst.multi INFO: found job 16889826 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 163 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 446c2059d7808563d9884f324f9b0477 with result from task 163 parsl.dataflow.dflow DEBUG: Task 163 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 164 has memoization hash 54ff5938b02c8e0e2baac52b0ec02074 parsl.dataflow.memoization INFO: Task 164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eafc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 409 parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/24c91ae1-5718-4b1b-a4a9-ed2526abef2b_calibrate_1125542_35.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/24c91ae1-5718-4b1b-a4a9-ed2526abef2b_calibrate_1125542_35.stderr parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/26224822-8c7b-4190-8099-92c0eed530ab_characterizeImage_1125542_35.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/26224822-8c7b-4190-8099-92c0eed530ab_characterizeImage_1125542_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:28 proc_lsst.multi:146 [INFO] found job 16889827 in provider astro proc_lsst.multi INFO: found job 16889827 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 34 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1bf438d44df45659d7376f68bd0b2067 with result from task 34 parsl.dataflow.dflow DEBUG: Task 34 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 35 has memoization hash ec78bc8912a528bd955db41f0e845712 parsl.dataflow.memoization INFO: Task 35 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134932e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 410 parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/f0c8e90c-75c7-4d69-a4a0-2b57b4799dcb_calibrate_1125543_15.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/f0c8e90c-75c7-4d69-a4a0-2b57b4799dcb_calibrate_1125543_15.stderr parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/19b54eb2-49be-480c-ab6e-a375d2ffbd56_characterizeImage_1125543_15.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/19b54eb2-49be-480c-ab6e-a375d2ffbd56_characterizeImage_1125543_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:29 proc_lsst.multi:146 [INFO] found job 16889828 in provider astro proc_lsst.multi INFO: found job 16889828 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 76 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d815d6cfcc611b37aa0f7f4afa1a39f9 with result from task 76 parsl.dataflow.dflow DEBUG: Task 76 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 77 has memoization hash 219df6d39d828ae8c630907f3e4e77bc parsl.dataflow.memoization INFO: Task 77 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b6160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 77 try 0 launched on executor multi with executor id 411 parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/2d19fb1f-bb11-496b-9059-280f8d67c19a_calibrate_1125543_35.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/2d19fb1f-bb11-496b-9059-280f8d67c19a_calibrate_1125543_35.stderr parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/e35d2dbe-38b4-4ffd-bf7d-6a1ae16389f7_characterizeImage_1125543_35.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/e35d2dbe-38b4-4ffd-bf7d-6a1ae16389f7_characterizeImage_1125543_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:30 proc_lsst.multi:146 [INFO] found job 16889829 in provider astro proc_lsst.multi INFO: found job 16889829 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 82 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a06e2f9d4f6adcbf7a4b6ae9f2ed123 with result from task 82 parsl.dataflow.dflow DEBUG: Task 82 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 83 has memoization hash a16ca02291dc704b45a85d0098078b62 parsl.dataflow.memoization INFO: Task 83 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b67a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 412 parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/29cd6056-33ae-4847-9124-f714137a8d13_calibrate_1125542_34.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/29cd6056-33ae-4847-9124-f714137a8d13_calibrate_1125542_34.stderr parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/e44a4715-c657-4955-8fd6-061b07829c0c_characterizeImage_1125542_34.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/e44a4715-c657-4955-8fd6-061b07829c0c_characterizeImage_1125542_34.stderr parsl.dataflow.dflow INFO: Task 67 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26d75d41dba3db9d93446fb1bff3929f with result from task 67 parsl.dataflow.dflow DEBUG: Task 67 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 68 has memoization hash 2876b2e173111f99ef685c249dbc1943 parsl.dataflow.memoization INFO: Task 68 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b5800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 413 parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/15e7f0ac-14f8-4ef7-aa22-6223ebcab5db_calibrate_1125541_32.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/15e7f0ac-14f8-4ef7-aa22-6223ebcab5db_calibrate_1125541_32.stderr parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/24be0878-a9b3-4379-934e-9de37d70bf6d_characterizeImage_1125541_32.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/24be0878-a9b3-4379-934e-9de37d70bf6d_characterizeImage_1125541_32.stderr parsl.dataflow.dflow INFO: Task 70 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10b6181370e03485340a4c0fc05af8b1 with result from task 70 parsl.dataflow.dflow DEBUG: Task 70 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 71 has memoization hash b97419799a1c5f6c7093d8bbe18349fd parsl.dataflow.memoization INFO: Task 71 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b5b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 414 parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/333fb6c0-26ff-45a2-ac32-787a5f81fd74_calibrate_1125543_42.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/333fb6c0-26ff-45a2-ac32-787a5f81fd74_calibrate_1125543_42.stderr parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/6a8a9d36-e7fb-4f05-9d75-d051f4aa7324_characterizeImage_1125543_42.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/6a8a9d36-e7fb-4f05-9d75-d051f4aa7324_characterizeImage_1125543_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:31 proc_lsst.multi:146 [INFO] found job 16889830 in provider astro proc_lsst.multi INFO: found job 16889830 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:32 proc_lsst.multi:146 [INFO] found job 16889831 in provider astro proc_lsst.multi INFO: found job 16889831 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:33 proc_lsst.multi:146 [INFO] found job 16889832 in provider astro proc_lsst.multi INFO: found job 16889832 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 157 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 749ab336978ea0ba7cb6d0932472405d with result from task 157 parsl.dataflow.dflow DEBUG: Task 157 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 158 has memoization hash eaead28d4de9cdc4173de290705e7e53 parsl.dataflow.memoization INFO: Task 158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eaf600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 415 parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/0f8e6a00-21cf-465b-b1f7-9438757b38f9_calibrate_1125542_15.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/0f8e6a00-21cf-465b-b1f7-9438757b38f9_calibrate_1125542_15.stderr parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/1611386f-214e-4fef-b952-8af42d0f8f7e_characterizeImage_1125542_15.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/1611386f-214e-4fef-b952-8af42d0f8f7e_characterizeImage_1125542_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:34 proc_lsst.multi:146 [INFO] found job 16889833 in provider astro proc_lsst.multi INFO: found job 16889833 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:35 proc_lsst.multi:146 [INFO] found job 16889834 in provider astro proc_lsst.multi INFO: found job 16889834 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 79 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4237b936c424b26f8e1d9ad3d8d20592 with result from task 79 parsl.dataflow.dflow DEBUG: Task 79 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 80 has memoization hash 8bec3eb215d1bc03e9499e019edc770b parsl.dataflow.memoization INFO: Task 80 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b6480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 416 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/db1ab732-f56a-4f1b-a6f2-42cb1cc89c90_calibrate_1125541_20.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/db1ab732-f56a-4f1b-a6f2-42cb1cc89c90_calibrate_1125541_20.stderr parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/15a0a6de-2d5b-41be-ba2b-9a3874804a9f_characterizeImage_1125541_20.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/15a0a6de-2d5b-41be-ba2b-9a3874804a9f_characterizeImage_1125541_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:36 proc_lsst.multi:146 [INFO] found job 16889835 in provider astro proc_lsst.multi INFO: found job 16889835 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 133 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d86fbf323b9cbea6783ab70a593aca82 with result from task 133 parsl.dataflow.dflow DEBUG: Task 133 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 134 has memoization hash 52b57bea44de99cde1d5f883ef199284 parsl.dataflow.memoization INFO: Task 134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eadd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 417 parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/f843776c-f938-404a-bf54-b9604651c914_calibrate_1125543_13.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/f843776c-f938-404a-bf54-b9604651c914_calibrate_1125543_13.stderr parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/a0c21ddd-0a50-45d2-a06a-eee8df892c9c_characterizeImage_1125543_13.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/a0c21ddd-0a50-45d2-a06a-eee8df892c9c_characterizeImage_1125543_13.stderr parsl.dataflow.dflow INFO: Task 130 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05a358fed292dfbc39ff02bf0936a6b4 with result from task 130 parsl.dataflow.dflow DEBUG: Task 130 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 131 has memoization hash 2f08debb9e3ed1e4c15ddca1daff8282 parsl.dataflow.memoization INFO: Task 131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ead9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 131 try 0 launched on executor multi with executor id 418 parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/74e6718b-4137-407d-afff-bdc071699184_calibrate_1125542_4.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/74e6718b-4137-407d-afff-bdc071699184_calibrate_1125542_4.stderr parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/1874074c-b59b-4af4-8aa9-2c00fe9a218f_characterizeImage_1125542_4.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/1874074c-b59b-4af4-8aa9-2c00fe9a218f_characterizeImage_1125542_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:38 proc_lsst.multi:146 [INFO] found job 16889836 in provider astro proc_lsst.multi INFO: found job 16889836 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:40 proc_lsst.multi:146 [INFO] found job 16889837 in provider astro proc_lsst.multi INFO: found job 16889837 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 4 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98c8131b984ca90f4f08919d2b36d7e5 with result from task 4 parsl.dataflow.dflow DEBUG: Task 4 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5 has memoization hash f69301463319e7df252348ee39648366 parsl.dataflow.memoization INFO: Task 5 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149813491440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 419 parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/04316636-23c7-443b-905d-36f142c34a44_calibrate_1125543_19.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/04316636-23c7-443b-905d-36f142c34a44_calibrate_1125543_19.stderr parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/a99fb97d-df94-40f1-9f84-d17465b8ff28_characterizeImage_1125543_19.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/a99fb97d-df94-40f1-9f84-d17465b8ff28_characterizeImage_1125543_19.stderr parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1aee49ed57402eaee37936fece0849f9 with result from task 115 parsl.dataflow.dflow DEBUG: Task 115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 116 has memoization hash f645c5e039dfb5a33b2777c64b22f40f parsl.dataflow.memoization INFO: Task 116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eaca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 420 parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/30c33530-89d6-4736-ab1e-bfbd5b200567_calibrate_1125542_20.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/30c33530-89d6-4736-ab1e-bfbd5b200567_calibrate_1125542_20.stderr parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/5d442e07-1f34-4834-8284-b06d63b0e90e_characterizeImage_1125542_20.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/5d442e07-1f34-4834-8284-b06d63b0e90e_characterizeImage_1125542_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:41 proc_lsst.multi:146 [INFO] found job 16889838 in provider astro proc_lsst.multi INFO: found job 16889838 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 166 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 924bdc8dbe00c6e7d2298aa9b0d930ec with result from task 166 parsl.dataflow.dflow DEBUG: Task 166 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 167 has memoization hash a4a17830d3bd6ddb528c9daa74b997ce parsl.dataflow.memoization INFO: Task 167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eafec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 167 try 0 launched on executor multi with executor id 421 parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/a0ac1c04-a38a-4aa8-83f9-8f62d2664245_calibrate_1125543_11.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/a0ac1c04-a38a-4aa8-83f9-8f62d2664245_calibrate_1125543_11.stderr parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/c211893b-c1c7-423f-87d2-62d3cdcfafe9_characterizeImage_1125543_11.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/c211893b-c1c7-423f-87d2-62d3cdcfafe9_characterizeImage_1125543_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:42 proc_lsst.multi:146 [INFO] found job 16889839 in provider astro proc_lsst.multi INFO: found job 16889839 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 136 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac8316b1c68b496916db94e912384d9b with result from task 136 parsl.dataflow.dflow DEBUG: Task 136 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 137 has memoization hash 8dc31811704d7682dea4cc8b74f84ef9 parsl.dataflow.memoization INFO: Task 137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eae020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 137 try 0 launched on executor multi with executor id 422 parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/c1dd1f8d-c12d-4c7e-a9ca-4bc75e2193ac_calibrate_1125543_24.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/c1dd1f8d-c12d-4c7e-a9ca-4bc75e2193ac_calibrate_1125543_24.stderr parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/aeb187d5-9384-4f86-938a-ab685e1478d3_characterizeImage_1125543_24.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/aeb187d5-9384-4f86-938a-ab685e1478d3_characterizeImage_1125543_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:42 proc_lsst.multi:146 [INFO] found job 16889840 in provider astro proc_lsst.multi INFO: found job 16889840 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:43 proc_lsst.multi:146 [INFO] found job 16889841 in provider astro proc_lsst.multi INFO: found job 16889841 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:44 proc_lsst.multi:146 [INFO] found job 16889842 in provider astro proc_lsst.multi INFO: found job 16889842 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:45 proc_lsst.multi:146 [INFO] found job 16889843 in provider astro proc_lsst.multi INFO: found job 16889843 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:46 proc_lsst.multi:146 [INFO] found job 16889844 in provider astro proc_lsst.multi INFO: found job 16889844 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 154 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 008f66b8352491cf792d78488331b3f3 with result from task 154 parsl.dataflow.dflow DEBUG: Task 154 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 155 has memoization hash e6c87fa34524dc46cbf9f72dae9b1972 parsl.dataflow.memoization INFO: Task 155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eaf2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 155 try 0 launched on executor multi with executor id 423 parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/f9aaa9db-8f88-4f3f-ae5a-552c020827e8_calibrate_1125543_41.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/f9aaa9db-8f88-4f3f-ae5a-552c020827e8_calibrate_1125543_41.stderr parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/a80d86c0-f901-48d0-a2e5-6db335450326_characterizeImage_1125543_41.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/a80d86c0-f901-48d0-a2e5-6db335450326_characterizeImage_1125543_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:47 proc_lsst.multi:146 [INFO] found job 16889845 in provider astro proc_lsst.multi INFO: found job 16889845 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 151 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65c99d32d9d599c39678eb42f2337216 with result from task 151 parsl.dataflow.dflow DEBUG: Task 151 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 152 has memoization hash eb7feaaa86e00b59040f7f1d8e51012d parsl.dataflow.memoization INFO: Task 152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eaefc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 424 parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/f12f82d5-0c88-45f9-825c-58c34ae413e3_calibrate_1125542_7.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/f12f82d5-0c88-45f9-825c-58c34ae413e3_calibrate_1125542_7.stderr parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/6bde3c23-0c93-4e91-8f1a-10a4e90db362_characterizeImage_1125542_7.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/6bde3c23-0c93-4e91-8f1a-10a4e90db362_characterizeImage_1125542_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:48 proc_lsst.multi:146 [INFO] found job 16889846 in provider astro proc_lsst.multi INFO: found job 16889846 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:49 proc_lsst.multi:146 [INFO] found job 16889847 in provider astro proc_lsst.multi INFO: found job 16889847 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 172 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93c51288c830dff9ee1171a649ad493a with result from task 172 parsl.dataflow.dflow DEBUG: Task 172 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 173 has memoization hash 9ef270fd3c3d19478c8827c0e837b04a parsl.dataflow.memoization INFO: Task 173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed45e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 173 try 0 launched on executor multi with executor id 425 parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/16121d9e-6592-4287-b0ff-10f383e5df76_calibrate_1125541_35.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/16121d9e-6592-4287-b0ff-10f383e5df76_calibrate_1125541_35.stderr parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/5735d9d8-af1e-4b4e-89fe-20a8b0b59b85_characterizeImage_1125541_35.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/5735d9d8-af1e-4b4e-89fe-20a8b0b59b85_characterizeImage_1125541_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:50 proc_lsst.multi:146 [INFO] found job 16889848 in provider astro proc_lsst.multi INFO: found job 16889848 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:51 proc_lsst.multi:146 [INFO] found job 16889852 in provider astro proc_lsst.multi INFO: found job 16889852 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:52 proc_lsst.multi:146 [INFO] found job 16889853 in provider astro proc_lsst.multi INFO: found job 16889853 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 193 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry afd65580dc2c6b34c53b59dd9f2dc160 with result from task 193 parsl.dataflow.dflow DEBUG: Task 193 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 194 has memoization hash 961b4f991e3ad4ef14def08af6fa4748 parsl.dataflow.memoization INFO: Task 194 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed5bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 194 try 0 launched on executor multi with executor id 426 parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/318eb8ea-25f0-41f8-812a-0bd380e2ea03_calibrate_1125542_13.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/318eb8ea-25f0-41f8-812a-0bd380e2ea03_calibrate_1125542_13.stderr parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/25d49a1c-4123-438f-ba15-c333cfa6face_characterizeImage_1125542_13.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/25d49a1c-4123-438f-ba15-c333cfa6face_characterizeImage_1125542_13.stderr parsl.dataflow.dflow INFO: Task 175 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 365991100e5a48cef498eb3573764c15 with result from task 175 parsl.dataflow.dflow DEBUG: Task 175 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 176 has memoization hash 498d1c97f3a7f329c1ca109c5afdf067 parsl.dataflow.memoization INFO: Task 176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed4900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 427 parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/c83f31ed-074a-414a-9016-3d2eeeb5cd12_calibrate_1125543_6.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/c83f31ed-074a-414a-9016-3d2eeeb5cd12_calibrate_1125543_6.stderr parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/7d3f2651-e638-42e3-8150-39a6621e78ff_characterizeImage_1125543_6.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/7d3f2651-e638-42e3-8150-39a6621e78ff_characterizeImage_1125543_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:53 proc_lsst.multi:146 [INFO] found job 16889854 in provider astro proc_lsst.multi INFO: found job 16889854 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 205 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f65a3cd949db2532bfbac6fc3dae2fdc with result from task 205 parsl.dataflow.dflow DEBUG: Task 205 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 206 has memoization hash 15d13a98d5fda6684c53f94371f1a49c parsl.dataflow.memoization INFO: Task 206 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed6840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 206 try 0 launched on executor multi with executor id 428 parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/df9ed02b-10f5-45e1-8e08-aadfba5612f3_calibrate_1125542_51.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/df9ed02b-10f5-45e1-8e08-aadfba5612f3_calibrate_1125542_51.stderr parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/759b503c-1c4f-4fa8-af74-51ee8573ed0e_characterizeImage_1125542_51.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/759b503c-1c4f-4fa8-af74-51ee8573ed0e_characterizeImage_1125542_51.stderr parsl.dataflow.dflow INFO: Task 118 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 186f55b18a0a62e4d55dd6398ab61a9c with result from task 118 parsl.dataflow.dflow DEBUG: Task 118 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 119 has memoization hash ce95901752377b691a8cbd94097c6ae7 parsl.dataflow.memoization INFO: Task 119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eacd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 119 try 0 launched on executor multi with executor id 429 parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/e90895ce-cfa1-49df-9e51-8dca3b9269af_calibrate_1125542_52.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/e90895ce-cfa1-49df-9e51-8dca3b9269af_calibrate_1125542_52.stderr parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/2f582667-1dba-4753-882c-6641b0f6f8a8_characterizeImage_1125542_52.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/2f582667-1dba-4753-882c-6641b0f6f8a8_characterizeImage_1125542_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:54 proc_lsst.multi:146 [INFO] found job 16889855 in provider astro proc_lsst.multi INFO: found job 16889855 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 211 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 079c51588cd1a77f894ddd59eff087a6 with result from task 211 parsl.dataflow.dflow DEBUG: Task 211 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 212 has memoization hash 65f7137ec394cc774e0e65656e0e6fec parsl.dataflow.memoization INFO: Task 212 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed6e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 212 try 0 launched on executor multi with executor id 430 parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/424fad7b-d73c-4c6b-9809-d4ee00f184bd_calibrate_1125542_29.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/424fad7b-d73c-4c6b-9809-d4ee00f184bd_calibrate_1125542_29.stderr parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/cd673753-b560-41f0-81d8-b8ff70e6d79a_characterizeImage_1125542_29.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/cd673753-b560-41f0-81d8-b8ff70e6d79a_characterizeImage_1125542_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:55 proc_lsst.multi:146 [INFO] found job 16889856 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889856 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.dataflow.dflow INFO: Task 73 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba940a1a5ae9ebc482b45df8c242ad59 with result from task 73 parsl.dataflow.dflow DEBUG: Task 73 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 74 has memoization hash a28d98b8e73c123f162849ebea6b2a93 parsl.dataflow.memoization INFO: Task 74 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1498134b5e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 431 parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/28035d15-a736-4672-b373-54304f1d23c8_calibrate_1125541_12.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/28035d15-a736-4672-b373-54304f1d23c8_calibrate_1125541_12.stderr parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/c8980db3-776f-498e-aaf7-f58046145065_characterizeImage_1125541_12.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/c8980db3-776f-498e-aaf7-f58046145065_characterizeImage_1125541_12.stderr parsl.dataflow.dflow INFO: Task 244 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 011d9db9848a972549587695ae5cc352 with result from task 244 parsl.dataflow.dflow DEBUG: Task 244 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 245 has memoization hash 44bbc5c5c58794ca9a0c2dacdde4e620 parsl.dataflow.memoization INFO: Task 245 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efd120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 245 try 0 launched on executor multi with executor id 432 parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/0399e33c-846a-4eed-b7fd-6f6908afb2a2_calibrate_1125542_6.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/0399e33c-846a-4eed-b7fd-6f6908afb2a2_calibrate_1125542_6.stderr parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/9ddb77ec-8e17-4193-bd7a-98ff07d6b69e_characterizeImage_1125542_6.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/9ddb77ec-8e17-4193-bd7a-98ff07d6b69e_characterizeImage_1125542_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:57 proc_lsst.multi:146 [INFO] found job 16889857 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889857 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.dataflow.dflow INFO: Task 247 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2dcd7fa3e418cf90d493424b5de1af03 with result from task 247 parsl.dataflow.dflow DEBUG: Task 247 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 248 has memoization hash d1c682868af8c561ab99258a4fa29dd3 parsl.dataflow.memoization INFO: Task 248 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efd440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 248 try 0 launched on executor multi with executor id 433 parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/b7cb758b-872e-4338-88af-b9b196345b40_calibrate_1125542_28.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/b7cb758b-872e-4338-88af-b9b196345b40_calibrate_1125542_28.stderr parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/1689aa09-62b1-43bc-9ce1-e26c20d8ca73_characterizeImage_1125542_28.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/1689aa09-62b1-43bc-9ce1-e26c20d8ca73_characterizeImage_1125542_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:58 proc_lsst.multi:146 [INFO] found job 16889858 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889858 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.dataflow.dflow INFO: Task 199 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3bbff10f3552cc00cbe64ece77f7dcc8 with result from task 199 parsl.dataflow.dflow DEBUG: Task 199 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 200 has memoization hash bd217515cd6ddbfd4301d0ce827d7a97 parsl.dataflow.memoization INFO: Task 200 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed6200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 200 try 0 launched on executor multi with executor id 434 parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/131b9c3c-38ae-4ccd-b411-5a40952171b9_calibrate_1125543_43.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/131b9c3c-38ae-4ccd-b411-5a40952171b9_calibrate_1125543_43.stderr parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/81444af2-1fcc-4ec4-88f6-9c30ec0ba60b_characterizeImage_1125543_43.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/81444af2-1fcc-4ec4-88f6-9c30ec0ba60b_characterizeImage_1125543_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:25:59 proc_lsst.multi:146 [INFO] found job 16889859 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889859 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:00 proc_lsst.multi:146 [INFO] found job 16889860 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889860 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:01 proc_lsst.multi:146 [INFO] found job 16889861 in provider escience_a40 proc_lsst.multi INFO: found job 16889861 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:02 proc_lsst.multi:146 [INFO] found job 16889862 in provider escience_a40 proc_lsst.multi INFO: found job 16889862 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:03 proc_lsst.multi:146 [INFO] found job 16889863 in provider escience_a40 proc_lsst.multi INFO: found job 16889863 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 142 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8be8e96bfb3ba4f0633d609e3e2aba44 with result from task 142 parsl.dataflow.dflow DEBUG: Task 142 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 143 has memoization hash 54af87744a00b01797e89cb17b7f399f parsl.dataflow.memoization INFO: Task 143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eae660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 143 try 0 launched on executor multi with executor id 435 parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/206123ad-9007-4aee-9724-7082099cc700_calibrate_1125543_18.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/206123ad-9007-4aee-9724-7082099cc700_calibrate_1125543_18.stderr parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/25ba8335-b796-4c59-8f38-a05a0632cabd_characterizeImage_1125543_18.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/25ba8335-b796-4c59-8f38-a05a0632cabd_characterizeImage_1125543_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:04 proc_lsst.multi:146 [INFO] found job 16889864 in provider escience_a40 proc_lsst.multi INFO: found job 16889864 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 262 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c10d10b8fb2759ffa57ebb9beecccf98 with result from task 262 parsl.dataflow.dflow DEBUG: Task 262 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 263 has memoization hash 426a47c9979f968cccd87eb8daa3ea75 parsl.dataflow.memoization INFO: Task 263 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efe3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 263 try 0 launched on executor multi with executor id 436 parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/e083836a-e28f-4f05-88fc-42dd3074a783_calibrate_1125541_9.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/e083836a-e28f-4f05-88fc-42dd3074a783_calibrate_1125541_9.stderr parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/6dd8c99c-afdd-495d-a62d-a8d1df8e0a12_characterizeImage_1125541_9.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/6dd8c99c-afdd-495d-a62d-a8d1df8e0a12_characterizeImage_1125541_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:05 proc_lsst.multi:146 [INFO] found job 16889865 in provider escience_a40 proc_lsst.multi INFO: found job 16889865 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:06 proc_lsst.multi:146 [INFO] found job 16889866 in provider escience_a40 proc_lsst.multi INFO: found job 16889866 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:08 proc_lsst.multi:146 [INFO] found job 16889867 in provider escience_a40 proc_lsst.multi INFO: found job 16889867 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 178 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 430ff1c8c5655c2f6b5c91c63fcbdb70 with result from task 178 parsl.dataflow.dflow DEBUG: Task 178 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 179 has memoization hash d288afe46a9d414c230b708bbcc8810f parsl.dataflow.memoization INFO: Task 179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed4c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 179 try 0 launched on executor multi with executor id 437 parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/ec05b8dc-753e-4fd4-8f2c-88a18cb78671_calibrate_1125542_10.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/ec05b8dc-753e-4fd4-8f2c-88a18cb78671_calibrate_1125542_10.stderr parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/72ddccf0-37eb-49fd-8fc8-c023534a94bb_characterizeImage_1125542_10.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/72ddccf0-37eb-49fd-8fc8-c023534a94bb_characterizeImage_1125542_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:09 proc_lsst.multi:146 [INFO] found job 16889868 in provider escience_a40 proc_lsst.multi INFO: found job 16889868 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 271 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b4fa9ba9b3e9b206e9e92f3f92d8aa3 with result from task 271 parsl.dataflow.dflow DEBUG: Task 271 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 272 has memoization hash 9f68f8d5d74b78476d1f274f04dd64cf parsl.dataflow.memoization INFO: Task 272 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 272 try 0 launched on executor multi with executor id 438 parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/b21a8ab8-22d7-4bf0-b688-2afd75ac55d3_calibrate_1125543_36.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/b21a8ab8-22d7-4bf0-b688-2afd75ac55d3_calibrate_1125543_36.stderr parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/328aa6fc-2464-450b-84f8-019707c925e4_characterizeImage_1125543_36.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/328aa6fc-2464-450b-84f8-019707c925e4_characterizeImage_1125543_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:12 proc_lsst.multi:146 [INFO] found job 16889869 in provider escience_a40 proc_lsst.multi INFO: found job 16889869 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 250 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 720056684c698991637df1ea74ba9ef6 with result from task 250 parsl.dataflow.dflow DEBUG: Task 250 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 251 has memoization hash 9122292bce84d98409a5c31e66ee368b parsl.dataflow.memoization INFO: Task 251 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efd760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 251 try 0 launched on executor multi with executor id 439 parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/0ca42d8b-bfc6-4ce6-baa6-b7c8c824c622_calibrate_1125542_49.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/0ca42d8b-bfc6-4ce6-baa6-b7c8c824c622_calibrate_1125542_49.stderr parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/f745f1ea-6776-4954-b769-4969bee4134c_characterizeImage_1125542_49.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/f745f1ea-6776-4954-b769-4969bee4134c_characterizeImage_1125542_49.stderr parsl.dataflow.dflow INFO: Task 184 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb5b1a5c88f948370d98b731bf77ed4d with result from task 184 parsl.dataflow.dflow DEBUG: Task 184 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 185 has memoization hash 3c1f1bc2aa758c0c7157935c7b244a54 parsl.dataflow.memoization INFO: Task 185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed5260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 185 try 0 launched on executor multi with executor id 440 parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/f9bdf75b-3953-44b5-8313-5f3b7c7832e2_calibrate_1125543_59.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/f9bdf75b-3953-44b5-8313-5f3b7c7832e2_calibrate_1125543_59.stderr parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/ce482e12-4df9-4d97-a9f1-02f1098cbd52_characterizeImage_1125543_59.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/ce482e12-4df9-4d97-a9f1-02f1098cbd52_characterizeImage_1125543_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:13 proc_lsst.multi:146 [INFO] found job 16889870 in provider escience_a40 proc_lsst.multi INFO: found job 16889870 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 256 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f57ce37445ce76d616822b391a444df with result from task 256 parsl.dataflow.dflow DEBUG: Task 256 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 257 has memoization hash 237daa15b91a8ab8c715b61e72e308a2 parsl.dataflow.memoization INFO: Task 257 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efdda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 257 try 0 launched on executor multi with executor id 441 parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/bf116c39-d646-4ab4-98dc-7bf5b62b4589_calibrate_1125542_9.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/bf116c39-d646-4ab4-98dc-7bf5b62b4589_calibrate_1125542_9.stderr parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/19d70ebb-e46e-4b12-9798-08bf16b4563b_characterizeImage_1125542_9.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/19d70ebb-e46e-4b12-9798-08bf16b4563b_characterizeImage_1125542_9.stderr parsl.dataflow.dflow INFO: Task 181 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e913f26274d93e4c16807668de2f8861 with result from task 181 parsl.dataflow.dflow DEBUG: Task 181 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 182 has memoization hash 18543d741b3cc0787362fae532b0e7f6 parsl.dataflow.memoization INFO: Task 182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed4f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 442 parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/394242a4-72fe-4b0c-9568-81ede1fc6638_calibrate_1125542_42.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/394242a4-72fe-4b0c-9568-81ede1fc6638_calibrate_1125542_42.stderr parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/e0f8c4d5-16b0-40c6-b955-271a8dc4ae7e_characterizeImage_1125542_42.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/e0f8c4d5-16b0-40c6-b955-271a8dc4ae7e_characterizeImage_1125542_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:14 proc_lsst.multi:146 [INFO] found job 16889871 in provider escience_a40 proc_lsst.multi INFO: found job 16889871 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22643663051856 2024-03-07 10:26:15 proc_lsst.multi:146 [INFO] found job 24403 in provider local proc_lsst.multi INFO: found job 24403 in provider local 2024-03-07 10:26:15 proc_lsst.multi:146 [INFO] found job 16889823 in provider astro proc_lsst.multi INFO: found job 16889823 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 190 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f9042a7e8c127a710b7b2260b6bc7bb with result from task 190 parsl.dataflow.dflow DEBUG: Task 190 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 191 has memoization hash c994feed6074735badc7c7025ca363c6 parsl.dataflow.memoization INFO: Task 191 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed58a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 191 try 0 launched on executor multi with executor id 443 parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/1c6e7fa1-aa9f-4296-88cb-a199a0700c7a_calibrate_1125543_57.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/1c6e7fa1-aa9f-4296-88cb-a199a0700c7a_calibrate_1125543_57.stderr parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/cf7737a9-f3ca-4323-9e0c-a41aa91f09c2_characterizeImage_1125543_57.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/cf7737a9-f3ca-4323-9e0c-a41aa91f09c2_characterizeImage_1125543_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:16 proc_lsst.multi:146 [INFO] found job 16889824 in provider astro proc_lsst.multi INFO: found job 16889824 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:17 proc_lsst.multi:146 [INFO] found job 16889825 in provider astro proc_lsst.multi INFO: found job 16889825 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 298 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0f29adcb47fb69ff6f6ea256dec343c with result from task 298 parsl.dataflow.dflow DEBUG: Task 298 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 299 has memoization hash fc0ac030e1da00991321daa3af83e165 parsl.dataflow.memoization INFO: Task 299 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f309a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 299 try 0 launched on executor multi with executor id 444 parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/ee5536ae-32cb-40e4-a48e-0af0c7465f4c_calibrate_1125543_31.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/ee5536ae-32cb-40e4-a48e-0af0c7465f4c_calibrate_1125543_31.stderr parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/dae85609-6d64-4adc-830b-410ea0b3c9ee_characterizeImage_1125543_31.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/dae85609-6d64-4adc-830b-410ea0b3c9ee_characterizeImage_1125543_31.stderr parsl.dataflow.dflow INFO: Task 217 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f5621cc4fd8a613e972afb438b3c0a0 with result from task 217 parsl.dataflow.dflow DEBUG: Task 217 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 218 has memoization hash 91e45dd03b52698a0f5e1947c49b2a4e parsl.dataflow.memoization INFO: Task 218 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed74c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 218 try 0 launched on executor multi with executor id 445 parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/caabbf03-6115-4c8f-912b-6616a6f8685c_calibrate_1125542_32.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/caabbf03-6115-4c8f-912b-6616a6f8685c_calibrate_1125542_32.stderr parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/0b574e50-53da-4078-a053-5f85c9e112ff_characterizeImage_1125542_32.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/0b574e50-53da-4078-a053-5f85c9e112ff_characterizeImage_1125542_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:18 proc_lsst.multi:146 [INFO] found job 16889826 in provider astro proc_lsst.multi INFO: found job 16889826 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:19 proc_lsst.multi:146 [INFO] found job 16889827 in provider astro proc_lsst.multi INFO: found job 16889827 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 214 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b47b7007b1bccb5e97da513b9fe67e06 with result from task 214 parsl.dataflow.dflow DEBUG: Task 214 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 215 has memoization hash 9fe7d2aef32b9b38ef0c39b78c8b5ac5 parsl.dataflow.memoization INFO: Task 215 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed71a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 215 try 0 launched on executor multi with executor id 446 parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/0a261d90-affc-40cd-834b-403b14146c44_calibrate_1125542_48.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/0a261d90-affc-40cd-834b-403b14146c44_calibrate_1125542_48.stderr parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/72f18c80-f999-4d22-8652-6112c3d01f0a_characterizeImage_1125542_48.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/72f18c80-f999-4d22-8652-6112c3d01f0a_characterizeImage_1125542_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:20 proc_lsst.multi:146 [INFO] found job 16889828 in provider astro proc_lsst.multi INFO: found job 16889828 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:21 proc_lsst.multi:146 [INFO] found job 16889829 in provider astro proc_lsst.multi INFO: found job 16889829 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:22 proc_lsst.multi:146 [INFO] found job 16889830 in provider astro proc_lsst.multi INFO: found job 16889830 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:23 proc_lsst.multi:146 [INFO] found job 16889831 in provider astro proc_lsst.multi INFO: found job 16889831 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 187 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91ec1e87085da9718657eee1cb447bff with result from task 187 parsl.dataflow.dflow DEBUG: Task 187 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 188 has memoization hash ef913d798e94881b2a92e15d944aedf9 parsl.dataflow.memoization INFO: Task 188 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed5580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 188 try 0 launched on executor multi with executor id 447 parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/411ac3a9-2943-4589-9710-a2b7480f0557_calibrate_1125541_43.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/411ac3a9-2943-4589-9710-a2b7480f0557_calibrate_1125541_43.stderr parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/6aa20301-9608-4428-b4af-880920b11115_characterizeImage_1125541_43.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/6aa20301-9608-4428-b4af-880920b11115_characterizeImage_1125541_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:24 proc_lsst.multi:146 [INFO] found job 16889832 in provider astro proc_lsst.multi INFO: found job 16889832 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:25 proc_lsst.multi:146 [INFO] found job 16889833 in provider astro proc_lsst.multi INFO: found job 16889833 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:26 proc_lsst.multi:146 [INFO] found job 16889834 in provider astro proc_lsst.multi INFO: found job 16889834 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 304 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd1c321c5b72eeb7fb5bdf0cc93e5959 with result from task 304 parsl.dataflow.dflow DEBUG: Task 304 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 305 has memoization hash 10ef93e188ae966b722adab247735c10 parsl.dataflow.memoization INFO: Task 305 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f30fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 305 try 0 launched on executor multi with executor id 448 parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/a4b1b3b7-215c-41da-8c2f-1dc377d36ac1_calibrate_1125541_46.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/a4b1b3b7-215c-41da-8c2f-1dc377d36ac1_calibrate_1125541_46.stderr parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/8ad48c45-e8b4-46d5-ad90-820e3701b332_characterizeImage_1125541_46.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/8ad48c45-e8b4-46d5-ad90-820e3701b332_characterizeImage_1125541_46.stderr parsl.dataflow.dflow INFO: Task 283 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5aec5e8b19c9ed9a4537348f90355ba with result from task 283 parsl.dataflow.dflow DEBUG: Task 283 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 284 has memoization hash 0965391c46366213450fef1358614414 parsl.dataflow.memoization INFO: Task 284 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eff9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 284 try 0 launched on executor multi with executor id 449 parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/70af7927-8e58-4621-b672-ba357ff91759_calibrate_1125543_7.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/70af7927-8e58-4621-b672-ba357ff91759_calibrate_1125543_7.stderr parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/51869941-58e3-4045-ab43-1c9b2fa1b986_characterizeImage_1125543_7.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/51869941-58e3-4045-ab43-1c9b2fa1b986_characterizeImage_1125543_7.stderr parsl.dataflow.dflow INFO: Task 280 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1a8628a0929772a1e864656a33f395b with result from task 280 parsl.dataflow.dflow DEBUG: Task 280 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 281 has memoization hash c38778fe7f103a71946258cb137856b5 parsl.dataflow.memoization INFO: Task 281 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eff6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 281 try 0 launched on executor multi with executor id 450 parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/b57dd2bd-ffef-4760-887b-cbd7e7b96acc_calibrate_1125543_54.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/b57dd2bd-ffef-4760-887b-cbd7e7b96acc_calibrate_1125543_54.stderr parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/7dc6a1a2-0c59-47c1-bef0-c1e84108dbc3_characterizeImage_1125543_54.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/7dc6a1a2-0c59-47c1-bef0-c1e84108dbc3_characterizeImage_1125543_54.stderr parsl.dataflow.dflow INFO: Task 322 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ead9ba522a6fdc834759310cb29ad72f with result from task 322 parsl.dataflow.dflow DEBUG: Task 322 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 323 has memoization hash 605fc10a72295dd137beba6b9305ab0f parsl.dataflow.memoization INFO: Task 323 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f322a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 323 try 0 launched on executor multi with executor id 451 parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/5cdaec8a-c14d-414b-8fc7-95b1f69fefc6_calibrate_1125541_36.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/5cdaec8a-c14d-414b-8fc7-95b1f69fefc6_calibrate_1125541_36.stderr parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/262ccbcd-d805-4531-ad67-0bf24c899881_characterizeImage_1125541_36.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/262ccbcd-d805-4531-ad67-0bf24c899881_characterizeImage_1125541_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:27 proc_lsst.multi:146 [INFO] found job 16889835 in provider astro proc_lsst.multi INFO: found job 16889835 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:28 proc_lsst.multi:146 [INFO] found job 16889836 in provider astro proc_lsst.multi INFO: found job 16889836 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 292 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9eed7683fd4318a9675253666c4e290 with result from task 292 parsl.dataflow.dflow DEBUG: Task 292 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 293 has memoization hash c1a6f169a21dda174ac721df66bf2b65 parsl.dataflow.memoization INFO: Task 293 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f30360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 293 try 0 launched on executor multi with executor id 452 parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/8a6fb022-1392-44a9-b134-7869e416d237_calibrate_1125541_55.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/8a6fb022-1392-44a9-b134-7869e416d237_calibrate_1125541_55.stderr parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/ff527bed-cfc7-41f4-b1c4-35443522f817_characterizeImage_1125541_55.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/ff527bed-cfc7-41f4-b1c4-35443522f817_characterizeImage_1125541_55.stderr parsl.dataflow.dflow INFO: Task 319 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f735f177d9a1a55671b2427c6d932807 with result from task 319 parsl.dataflow.dflow DEBUG: Task 319 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 320 has memoization hash 934ae968b13822a48e30f81b5ba64519 parsl.dataflow.memoization INFO: Task 320 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f31f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 320 try 0 launched on executor multi with executor id 453 parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/b13cde07-ef0f-4dab-b30e-b6bebec2b2c6_calibrate_1125541_31.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/b13cde07-ef0f-4dab-b30e-b6bebec2b2c6_calibrate_1125541_31.stderr parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/f64684ff-da22-42a8-a535-7b229c4af21c_characterizeImage_1125541_31.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/f64684ff-da22-42a8-a535-7b229c4af21c_characterizeImage_1125541_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:29 proc_lsst.multi:146 [INFO] found job 16889837 in provider astro proc_lsst.multi INFO: found job 16889837 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 208 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ada2b99af08dfeb37f0d465e28e405ca with result from task 208 parsl.dataflow.dflow DEBUG: Task 208 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 209 has memoization hash cc1a28468607ea902a5c39984d4ca151 parsl.dataflow.memoization INFO: Task 209 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed6b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 209 try 0 launched on executor multi with executor id 454 parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/da7f1ae9-85cd-4d1a-9370-1c238e977e29_calibrate_1125541_48.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/da7f1ae9-85cd-4d1a-9370-1c238e977e29_calibrate_1125541_48.stderr parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/fa0d9485-6711-4c37-8f88-5fe8c7e46838_characterizeImage_1125541_48.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/fa0d9485-6711-4c37-8f88-5fe8c7e46838_characterizeImage_1125541_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:31 proc_lsst.multi:146 [INFO] found job 16889838 in provider astro proc_lsst.multi INFO: found job 16889838 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 301 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8829ccc8e3a2831c35a1770b5451c033 with result from task 301 parsl.dataflow.dflow DEBUG: Task 301 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 302 has memoization hash 4e0014a2f0b6e668fc2314d19e851c37 parsl.dataflow.memoization INFO: Task 302 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f30cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 302 try 0 launched on executor multi with executor id 455 parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/1d97e5ad-fc93-42e6-a0ce-1badf1be756c_calibrate_1125543_22.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/1d97e5ad-fc93-42e6-a0ce-1badf1be756c_calibrate_1125543_22.stderr parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/81432c41-a551-45bd-9a30-34db2866f125_characterizeImage_1125543_22.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/81432c41-a551-45bd-9a30-34db2866f125_characterizeImage_1125543_22.stderr parsl.dataflow.dflow INFO: Task 310 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 929b477ca7acc2368346b7499ff27a4a with result from task 310 parsl.dataflow.dflow DEBUG: Task 310 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 311 has memoization hash 8fc594c06499d832b000484d47623742 parsl.dataflow.memoization INFO: Task 311 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f31620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 311 try 0 launched on executor multi with executor id 456 parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/7743cf93-9e5e-4387-97b8-d483a8d6d0db_calibrate_1125542_16.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/7743cf93-9e5e-4387-97b8-d483a8d6d0db_calibrate_1125542_16.stderr parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/1691dedf-2669-4324-baf6-78ba005df4e7_characterizeImage_1125542_16.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/1691dedf-2669-4324-baf6-78ba005df4e7_characterizeImage_1125542_16.stderr parsl.dataflow.dflow INFO: Task 202 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a98fa7dc46b41b4697f47f0ca2f6b3c with result from task 202 parsl.dataflow.dflow DEBUG: Task 202 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 203 has memoization hash 93c330dd4d5a298f11fdf45146b24a00 parsl.dataflow.memoization INFO: Task 203 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed6520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 203 try 0 launched on executor multi with executor id 457 parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/2022f50a-96b2-4619-bbbe-a65f1d1bba21_calibrate_1125543_56.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/2022f50a-96b2-4619-bbbe-a65f1d1bba21_calibrate_1125543_56.stderr parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/65a5853e-3bc2-4eac-843d-96d0f7414f4d_characterizeImage_1125543_56.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/65a5853e-3bc2-4eac-843d-96d0f7414f4d_characterizeImage_1125543_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:32 proc_lsst.multi:146 [INFO] found job 16889839 in provider astro proc_lsst.multi INFO: found job 16889839 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 268 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e9c8af4541d2b73e158d58bf1306a35 with result from task 268 parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:33 proc_lsst.multi:146 [INFO] found job 16889840 in provider astro proc_lsst.multi INFO: found job 16889840 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow DEBUG: Task 268 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 269 has memoization hash 155be302b59b5704e536c2539e3656d2 parsl.dataflow.memoization INFO: Task 269 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 269 try 0 launched on executor multi with executor id 458 parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/a18f5bd3-fb64-48e8-8099-22aada8ddc61_calibrate_1125542_25.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/a18f5bd3-fb64-48e8-8099-22aada8ddc61_calibrate_1125542_25.stderr parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/4b9e11f7-6e2f-4605-9a8b-4e084d8d0caa_characterizeImage_1125542_25.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/4b9e11f7-6e2f-4605-9a8b-4e084d8d0caa_characterizeImage_1125542_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:33 proc_lsst.multi:146 [INFO] found job 16889841 in provider astro proc_lsst.multi INFO: found job 16889841 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 328 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4aff324b0b49346d5ee18d13bf216dd with result from task 328 parsl.dataflow.dflow DEBUG: Task 328 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 329 has memoization hash 6e97c6a3c4a52348e6887d63996db333 parsl.dataflow.memoization INFO: Task 329 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f328e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 329 try 0 launched on executor multi with executor id 459 parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/b8a846d0-65e8-45a6-98fe-1794b863e012_calibrate_1125541_41.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/b8a846d0-65e8-45a6-98fe-1794b863e012_calibrate_1125541_41.stderr parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/b2878146-f284-4dc5-a2e5-b525e5c594dc_characterizeImage_1125541_41.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/b2878146-f284-4dc5-a2e5-b525e5c594dc_characterizeImage_1125541_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:34 proc_lsst.multi:146 [INFO] found job 16889842 in provider astro proc_lsst.multi INFO: found job 16889842 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:35 proc_lsst.multi:146 [INFO] found job 16889843 in provider astro proc_lsst.multi INFO: found job 16889843 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 274 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14bdd080c84f72d7ac1643075aa547e3 with result from task 274 parsl.dataflow.dflow DEBUG: Task 274 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 275 has memoization hash 7daa86b1344cad050e5e3d55b648dd18 parsl.dataflow.memoization INFO: Task 275 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eff060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 275 try 0 launched on executor multi with executor id 460 parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/b1695b08-1308-4b3a-a428-b9d29750a46b_calibrate_1125541_24.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/b1695b08-1308-4b3a-a428-b9d29750a46b_calibrate_1125541_24.stderr parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/5ff71989-ed8c-437b-bc33-5da799703f53_characterizeImage_1125541_24.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/5ff71989-ed8c-437b-bc33-5da799703f53_characterizeImage_1125541_24.stderr parsl.dataflow.dflow INFO: Task 325 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bcbb9d6d514cfc7d16d6b0b64539ce22 with result from task 325 parsl.dataflow.dflow DEBUG: Task 325 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 326 has memoization hash f7e5a9d29e460fc2733bf16434a495c8 parsl.dataflow.memoization INFO: Task 326 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f325c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 326 try 0 launched on executor multi with executor id 461 parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/9d6fbc77-8df6-4890-8538-356bd01e68cb_calibrate_1125542_36.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/9d6fbc77-8df6-4890-8538-356bd01e68cb_calibrate_1125542_36.stderr parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/b274198d-ef9f-44d5-b778-c264ecf488e5_characterizeImage_1125542_36.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/b274198d-ef9f-44d5-b778-c264ecf488e5_characterizeImage_1125542_36.stderr parsl.dataflow.dflow INFO: Task 289 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b68834df984e8c379657dd061e71ef91 with result from task 289 parsl.dataflow.dflow DEBUG: Task 289 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 290 has memoization hash ead3ab61c3103063384fbc82e410360e parsl.dataflow.memoization INFO: Task 290 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f30220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 290 try 0 launched on executor multi with executor id 462 parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/32c9971d-848d-4768-89ed-c5f0650a7531_calibrate_1125542_53.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/32c9971d-848d-4768-89ed-c5f0650a7531_calibrate_1125542_53.stderr parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/41fad328-bceb-4a92-a088-8e7e61452309_characterizeImage_1125542_53.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/41fad328-bceb-4a92-a088-8e7e61452309_characterizeImage_1125542_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:36 proc_lsst.multi:146 [INFO] found job 16889844 in provider astro proc_lsst.multi INFO: found job 16889844 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 343 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 320de90e2b1ff9d8d663fb0bef25ae7a with result from task 343 parsl.dataflow.dflow DEBUG: Task 343 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 344 has memoization hash 9ac16e6ad230784c0723abe530c5420a parsl.dataflow.memoization INFO: Task 344 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f33880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 344 try 0 launched on executor multi with executor id 463 parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/156ab4d8-4858-4963-8e9e-2bf2f18549f1_calibrate_1125541_19.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/156ab4d8-4858-4963-8e9e-2bf2f18549f1_calibrate_1125541_19.stderr parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/1baf5726-d1b3-44f1-9fa3-6a83de5ab826_characterizeImage_1125541_19.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/1baf5726-d1b3-44f1-9fa3-6a83de5ab826_characterizeImage_1125541_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:37 proc_lsst.multi:146 [INFO] found job 16889845 in provider astro proc_lsst.multi INFO: found job 16889845 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 385 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7004952a6f1c745e125807ebc5b5fc7d with result from task 385 parsl.dataflow.dflow DEBUG: Task 385 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 386 has memoization hash 658d023ae65a2b88a870e71e0a9741ec parsl.dataflow.memoization INFO: Task 386 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f7a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 386 try 0 launched on executor multi with executor id 464 parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/6f4db56a-ecc1-4d17-b7a2-77a57f5113b1_calibrate_1125542_57.stdout parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/6f4db56a-ecc1-4d17-b7a2-77a57f5113b1_calibrate_1125542_57.stderr parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/bf4b57dd-7fb3-4e1b-a196-4164dfe425fc_characterizeImage_1125542_57.stdout parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/bf4b57dd-7fb3-4e1b-a196-4164dfe425fc_characterizeImage_1125542_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:39 proc_lsst.multi:146 [INFO] found job 16889846 in provider astro proc_lsst.multi INFO: found job 16889846 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 358 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd3f49f6622a4bba8663f9acbbfbe2fa with result from task 358 parsl.dataflow.dflow DEBUG: Task 358 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 359 has memoization hash d44b9fe7c72e4ebb4960331480b58cae parsl.dataflow.memoization INFO: Task 359 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f78860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 359 try 0 launched on executor multi with executor id 465 parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/4d0a7e29-1c53-4085-9a20-9dad80c22c99_calibrate_1125542_47.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/4d0a7e29-1c53-4085-9a20-9dad80c22c99_calibrate_1125542_47.stderr parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/d6a8aed5-11ca-48a1-b29a-f05fc8f92d46_characterizeImage_1125542_47.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/d6a8aed5-11ca-48a1-b29a-f05fc8f92d46_characterizeImage_1125542_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:39 proc_lsst.multi:146 [INFO] found job 16889847 in provider astro proc_lsst.multi INFO: found job 16889847 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 355 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8b2306d8ee054dee80c9d11f780febb with result from task 355 parsl.dataflow.dflow DEBUG: Task 355 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 356 has memoization hash 77016307c3deba7d890ededbfa72c528 parsl.dataflow.memoization INFO: Task 356 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f78540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 356 try 0 launched on executor multi with executor id 466 parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/854171dc-f0b5-4c6d-8c16-274725e2ca40_calibrate_1125542_11.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/854171dc-f0b5-4c6d-8c16-274725e2ca40_calibrate_1125542_11.stderr parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/35adb483-89a5-4584-9817-8733569d4390_characterizeImage_1125542_11.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/35adb483-89a5-4584-9817-8733569d4390_characterizeImage_1125542_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:40 proc_lsst.multi:146 [INFO] found job 16889848 in provider astro proc_lsst.multi INFO: found job 16889848 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:41 proc_lsst.multi:146 [INFO] found job 16889852 in provider astro proc_lsst.multi INFO: found job 16889852 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 295 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5350aae8b892877fddd35e8b48614d06 with result from task 295 parsl.dataflow.dflow DEBUG: Task 295 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 296 has memoization hash f87da21828913efc2788ae01db6b6311 parsl.dataflow.memoization INFO: Task 296 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f30680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 296 try 0 launched on executor multi with executor id 467 parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/25a9c9f7-fd8f-44d7-be6e-830536fb0e9f_calibrate_1125541_14.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/25a9c9f7-fd8f-44d7-be6e-830536fb0e9f_calibrate_1125541_14.stderr parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/82472f56-debe-400f-95e1-7b7fe56aea03_characterizeImage_1125541_14.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/82472f56-debe-400f-95e1-7b7fe56aea03_characterizeImage_1125541_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:42 proc_lsst.multi:146 [INFO] found job 16889853 in provider astro proc_lsst.multi INFO: found job 16889853 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 259 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b0a5fd58c0c3a305c26eb82310d7542 with result from task 259 parsl.dataflow.dflow DEBUG: Task 259 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 260 has memoization hash 421820a9640a8871e710913cb080f90b parsl.dataflow.memoization INFO: Task 260 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efe0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 260 try 0 launched on executor multi with executor id 468 parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/cded50cc-d711-4b6e-82ca-4a0f22fbaba1_calibrate_1125543_58.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/cded50cc-d711-4b6e-82ca-4a0f22fbaba1_calibrate_1125543_58.stderr parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/91b32ed4-7ff0-45dc-813e-e245106f1baa_characterizeImage_1125543_58.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/91b32ed4-7ff0-45dc-813e-e245106f1baa_characterizeImage_1125543_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:45 proc_lsst.multi:146 [INFO] found job 16889854 in provider astro proc_lsst.multi INFO: found job 16889854 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 241 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4fc6208e805007c07f059b4888f05121 with result from task 241 parsl.dataflow.dflow DEBUG: Task 241 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 242 has memoization hash 32e8d46c707f8a5c72dcbe2c1f7ebdd4 parsl.dataflow.memoization INFO: Task 242 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 242 try 0 launched on executor multi with executor id 469 parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/080be549-6195-4e31-ac7c-19855332556d_calibrate_1125541_30.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/080be549-6195-4e31-ac7c-19855332556d_calibrate_1125541_30.stderr parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/5a60539a-df56-4e9c-9d92-01e10386fe02_characterizeImage_1125541_30.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/5a60539a-df56-4e9c-9d92-01e10386fe02_characterizeImage_1125541_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:46 proc_lsst.multi:146 [INFO] found job 16889855 in provider astro proc_lsst.multi INFO: found job 16889855 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:47 proc_lsst.multi:146 [INFO] found job 16889856 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889856 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.dataflow.dflow INFO: Task 331 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14b19c4a849405ae3ba140fb8cb0a810 with result from task 331 parsl.dataflow.dflow DEBUG: Task 331 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 332 has memoization hash fc976f8eecbf906a4109861056912737 parsl.dataflow.memoization INFO: Task 332 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f32c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 332 try 0 launched on executor multi with executor id 470 parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/caf2be89-e4af-4301-a01e-e197bafd21bf_calibrate_1125542_62.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/caf2be89-e4af-4301-a01e-e197bafd21bf_calibrate_1125542_62.stderr parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/f636359e-3add-4cf2-83b2-45514b25c8d5_characterizeImage_1125542_62.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/f636359e-3add-4cf2-83b2-45514b25c8d5_characterizeImage_1125542_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:48 proc_lsst.multi:146 [INFO] found job 16889857 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889857 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:49 proc_lsst.multi:146 [INFO] found job 16889858 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889858 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:50 proc_lsst.multi:146 [INFO] found job 16889859 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889859 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.dataflow.dflow INFO: Task 265 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18c0e625a89297d3afdb3bc430dbfaed with result from task 265 parsl.dataflow.dflow DEBUG: Task 265 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 266 has memoization hash 4acb2b5c6beed7b1a9f710385ea1e31a parsl.dataflow.memoization INFO: Task 266 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efe700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 266 try 0 launched on executor multi with executor id 471 parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/2f8a14e1-b362-439b-a847-c51df9891559_calibrate_1125542_17.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/2f8a14e1-b362-439b-a847-c51df9891559_calibrate_1125542_17.stderr parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/c70ccf96-444f-4fa7-93a4-da00461e04d7_characterizeImage_1125542_17.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/c70ccf96-444f-4fa7-93a4-da00461e04d7_characterizeImage_1125542_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:51 proc_lsst.multi:146 [INFO] found job 16889860 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889860 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.dataflow.dflow INFO: Task 361 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0891dbc7bdd25784d2229be6ce8cf3e0 with result from task 361 parsl.dataflow.dflow DEBUG: Task 361 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 362 has memoization hash e86a1f644a72548905cb7add3e2d45f5 parsl.dataflow.memoization INFO: Task 362 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f78b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 362 try 0 launched on executor multi with executor id 472 parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/b81d11b5-2d4e-44d7-8b70-83902b24b488_calibrate_1125543_10.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/b81d11b5-2d4e-44d7-8b70-83902b24b488_calibrate_1125543_10.stderr parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/dfcfdc1e-a07d-4168-b007-bc0c87754827_characterizeImage_1125543_10.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/dfcfdc1e-a07d-4168-b007-bc0c87754827_characterizeImage_1125543_10.stderr parsl.dataflow.dflow INFO: Task 337 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0a27f2bdc0f624ebf49c31d11777736 with result from task 337 parsl.dataflow.dflow DEBUG: Task 337 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 338 has memoization hash 601da629c1befe976e468c35b4456dc1 parsl.dataflow.memoization INFO: Task 338 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f33240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 338 try 0 launched on executor multi with executor id 473 parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/9502c041-ae42-4cd4-ba26-4e35f8eb5ba2_calibrate_1125543_3.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/9502c041-ae42-4cd4-ba26-4e35f8eb5ba2_calibrate_1125543_3.stderr parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/f7afc55c-ed96-43b9-9dcf-f9c5c202fefc_characterizeImage_1125543_3.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/f7afc55c-ed96-43b9-9dcf-f9c5c202fefc_characterizeImage_1125543_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:52 proc_lsst.multi:146 [INFO] found job 16889861 in provider escience_a40 proc_lsst.multi INFO: found job 16889861 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:53 proc_lsst.multi:146 [INFO] found job 16889862 in provider escience_a40 proc_lsst.multi INFO: found job 16889862 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:54 proc_lsst.multi:146 [INFO] found job 16889863 in provider escience_a40 proc_lsst.multi INFO: found job 16889863 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 253 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 812b6eaf4f31e9123bdabfbdf75bbe38 with result from task 253 parsl.dataflow.dflow DEBUG: Task 253 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 254 has memoization hash 116a4eb20e28b59e8938b4acf321d8d0 parsl.dataflow.memoization INFO: Task 254 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efda80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 254 try 0 launched on executor multi with executor id 474 parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/5b79a585-7777-43cd-82d3-29204b4df691_calibrate_1125541_49.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/5b79a585-7777-43cd-82d3-29204b4df691_calibrate_1125541_49.stderr parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/8483b0c1-d27d-429b-9c20-c96aa1a457e6_characterizeImage_1125541_49.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/8483b0c1-d27d-429b-9c20-c96aa1a457e6_characterizeImage_1125541_49.stderr parsl.dataflow.dflow INFO: Task 313 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dfecc735cb016ba7834a04a025b55913 with result from task 313 parsl.dataflow.dflow DEBUG: Task 313 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 314 has memoization hash 6766e81a31de330e6e11dae20eef02bf parsl.dataflow.memoization INFO: Task 314 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f31940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 314 try 0 launched on executor multi with executor id 475 parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/93f1fb2e-899e-4f1a-ae66-686bdad314f6_calibrate_1125541_17.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/93f1fb2e-899e-4f1a-ae66-686bdad314f6_calibrate_1125541_17.stderr parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/194fe1cf-aa35-45c1-88cf-f9622887344b_characterizeImage_1125541_17.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/194fe1cf-aa35-45c1-88cf-f9622887344b_characterizeImage_1125541_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:55 proc_lsst.multi:146 [INFO] found job 16889864 in provider escience_a40 proc_lsst.multi INFO: found job 16889864 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:56 proc_lsst.multi:146 [INFO] found job 16889865 in provider escience_a40 proc_lsst.multi INFO: found job 16889865 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:57 proc_lsst.multi:146 [INFO] found job 16889866 in provider escience_a40 proc_lsst.multi INFO: found job 16889866 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:58 proc_lsst.multi:146 [INFO] found job 16889867 in provider escience_a40 proc_lsst.multi INFO: found job 16889867 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:26:59 proc_lsst.multi:146 [INFO] found job 16889868 in provider escience_a40 proc_lsst.multi INFO: found job 16889868 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 376 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3381cac2f62a77db434de542e9734da9 with result from task 376 parsl.dataflow.dflow DEBUG: Task 376 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 377 has memoization hash 7beba6b4634de9ea4dba14ba4a0f7e22 parsl.dataflow.memoization INFO: Task 377 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f79b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 377 try 0 launched on executor multi with executor id 476 parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/b468c896-ba93-41d9-bbc2-29ccfbb419f6_calibrate_1125541_22.stdout parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/b468c896-ba93-41d9-bbc2-29ccfbb419f6_calibrate_1125541_22.stderr parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/f12171ac-6d29-4b1a-b40c-67b9658c5d34_characterizeImage_1125541_22.stdout parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/f12171ac-6d29-4b1a-b40c-67b9658c5d34_characterizeImage_1125541_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:27:00 proc_lsst.multi:146 [INFO] found job 16889869 in provider escience_a40 proc_lsst.multi INFO: found job 16889869 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:27:01 proc_lsst.multi:146 [INFO] found job 16889870 in provider escience_a40 proc_lsst.multi INFO: found job 16889870 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 400 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6723bb06593b7271777c7f1e55cc583 with result from task 400 parsl.dataflow.dflow DEBUG: Task 400 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 401 has memoization hash 0d212ae73f3712e896f29e58668b4b55 parsl.dataflow.memoization INFO: Task 401 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbdbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 401 try 0 launched on executor multi with executor id 477 parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/1379a7d9-9161-4898-81c2-3f929dcb1c76_calibrate_1125541_54.stdout parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/1379a7d9-9161-4898-81c2-3f929dcb1c76_calibrate_1125541_54.stderr parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/45c9f29a-ac2b-479b-a66a-1e9079f3bcc5_characterizeImage_1125541_54.stdout parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/45c9f29a-ac2b-479b-a66a-1e9079f3bcc5_characterizeImage_1125541_54.stderr parsl.dataflow.dflow INFO: Task 421 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bff74bed3f9b2167394fc251c4a9465e with result from task 421 parsl.dataflow.dflow DEBUG: Task 421 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 422 has memoization hash 83c46b5703f88a715b89300dd48d4558 parsl.dataflow.memoization INFO: Task 422 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f7ba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 422 try 0 launched on executor multi with executor id 478 parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/00e0ba00-d95d-41e2-873e-8ea0718fed80_calibrate_1125542_8.stdout parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/00e0ba00-d95d-41e2-873e-8ea0718fed80_calibrate_1125542_8.stderr parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/b87ae1b3-5089-43d9-a7e2-70db97c1e76e_characterizeImage_1125542_8.stdout parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/b87ae1b3-5089-43d9-a7e2-70db97c1e76e_characterizeImage_1125542_8.stderr parsl.dataflow.dflow INFO: Task 397 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17328f14d533e9b2da0e96d7d20aee48 with result from task 397 parsl.dataflow.dflow DEBUG: Task 397 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 398 has memoization hash e925a06bc2ac31c1759313f8a529a1b6 parsl.dataflow.memoization INFO: Task 398 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbdee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 398 try 0 launched on executor multi with executor id 479 parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/42c5b188-b3ee-412e-a0b5-fe41004cd7a5_calibrate_1125543_33.stdout parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/42c5b188-b3ee-412e-a0b5-fe41004cd7a5_calibrate_1125543_33.stderr parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/4f8ca024-7872-4a1a-bfd1-9c4999e35cff_characterizeImage_1125543_33.stdout parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/4f8ca024-7872-4a1a-bfd1-9c4999e35cff_characterizeImage_1125543_33.stderr parsl.dataflow.dflow INFO: Task 352 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9485271fe8568fe698882d79a86e3040 with result from task 352 parsl.dataflow.dflow DEBUG: Task 352 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 353 has memoization hash 82e3b293bbb074994630bf92d8b1432a parsl.dataflow.memoization INFO: Task 353 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f78220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 353 try 0 launched on executor multi with executor id 480 parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/8ec53b36-7310-48b0-b694-9f7c120b3999_calibrate_1125543_55.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/8ec53b36-7310-48b0-b694-9f7c120b3999_calibrate_1125543_55.stderr parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/82f41c71-8d8d-4b5f-9a34-43467ee84bd2_characterizeImage_1125543_55.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/82f41c71-8d8d-4b5f-9a34-43467ee84bd2_characterizeImage_1125543_55.stderr parsl.dataflow.dflow INFO: Task 406 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eda4f16cc74127de328f775f893d2169 with result from task 406 parsl.dataflow.dflow DEBUG: Task 406 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 407 has memoization hash 188ec5f686e7e35d154cc6a09f08e424 parsl.dataflow.memoization INFO: Task 407 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbd580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 407 try 0 launched on executor multi with executor id 481 parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/71cb81c5-2df9-4168-b88b-1e7591328d98_calibrate_1125541_16.stdout parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/71cb81c5-2df9-4168-b88b-1e7591328d98_calibrate_1125541_16.stderr parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/c8c937fa-8606-407b-a2a6-497f9e0815ea_characterizeImage_1125541_16.stdout parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/c8c937fa-8606-407b-a2a6-497f9e0815ea_characterizeImage_1125541_16.stderr parsl.channels.local.local WARNING: Execution of command 'squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871'' failed due to Command 'squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871'' timed out after 10 seconds parsl.utils ERROR: Callback threw an exception - logging and proceeding anyway Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/utils.py", line 341, in make_callback self.callback(*self.cb_args) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/job_status_poller.py", line 114, in poll self._update_state() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/job_status_poller.py", line 121, in _update_state item.poll(now) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/job_status_poller.py", line 47, in poll self._status = self._executor.status() ^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/executors/status_handling.py", line 109, in status status = self._make_status_dict(block_ids, self._provider.status(job_ids)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/gscratch/dirac/shared/opt/proc_lsst/python/proc_lsst/multi.py", line 190, in status statuses += self.providers[provider].status([job_id]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/providers/cluster_provider.py", line 142, in status self._status() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/providers/slurm/slurm.py", line 157, in _status retcode, stdout, stderr = self.execute_wait(cmd) ^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/providers/cluster_provider.py", line 83, in execute_wait return self.channel.execute_wait(cmd, t) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/channels/local/local.py", line 71, in execute_wait (stdout, stderr) = proc.communicate(timeout=walltime) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/subprocess.py", line 1209, in communicate stdout, stderr = self._communicate(input, endtime, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/subprocess.py", line 2114, in _communicate self._check_timeout(endtime, orig_timeout, stdout, stderr) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/subprocess.py", line 1253, in _check_timeout raise TimeoutExpired( subprocess.TimeoutExpired: Command 'squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871'' timed out after 10 seconds 2024-03-07 10:27:11 proc_lsst.multi:146 [INFO] found job 24403 in provider local proc_lsst.multi INFO: found job 24403 in provider local 2024-03-07 10:27:12 proc_lsst.multi:146 [INFO] found job 16889823 in provider astro proc_lsst.multi INFO: found job 16889823 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 364 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ac4c763e62a64cb76cebb6ecb86a067 with result from task 364 parsl.dataflow.dflow DEBUG: Task 364 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 365 has memoization hash e3135d0432eec4fcc28c91c8cf837ea7 parsl.dataflow.memoization INFO: Task 365 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f78ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 365 try 0 launched on executor multi with executor id 482 parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/051dce63-0194-4c6f-95ae-934f0ad1925d_calibrate_1125541_57.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/051dce63-0194-4c6f-95ae-934f0ad1925d_calibrate_1125541_57.stderr parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/fe37ebae-12e4-4b18-a319-ab390078232f_characterizeImage_1125541_57.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/fe37ebae-12e4-4b18-a319-ab390078232f_characterizeImage_1125541_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:27:13 proc_lsst.multi:146 [INFO] found job 16889824 in provider astro proc_lsst.multi INFO: found job 16889824 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 349 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30cf5bb14acf3c2b4f27f7b423d3a17c with result from task 349 parsl.dataflow.dflow DEBUG: Task 349 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 350 has memoization hash cd1b211c25987c30978d0b3018bf520d parsl.dataflow.memoization INFO: Task 350 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f33e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 350 try 0 launched on executor multi with executor id 483 parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/6a9e301d-6b5f-476b-aed4-ae9f8f583a19_calibrate_1125542_37.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/6a9e301d-6b5f-476b-aed4-ae9f8f583a19_calibrate_1125542_37.stderr parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/5f678f28-605d-4aec-9ff8-d4b815838c62_characterizeImage_1125542_37.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/5f678f28-605d-4aec-9ff8-d4b815838c62_characterizeImage_1125542_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:27:13 proc_lsst.multi:146 [INFO] found job 16889825 in provider astro proc_lsst.multi INFO: found job 16889825 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:27:14 proc_lsst.multi:146 [INFO] found job 16889826 in provider astro proc_lsst.multi INFO: found job 16889826 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 382 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry efa36736e35a1649b209502126d8da41 with result from task 382 parsl.dataflow.dflow DEBUG: Task 382 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 383 has memoization hash 09e813fe081f8d0504a75721ef50ceda parsl.dataflow.memoization INFO: Task 383 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f7a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 383 try 0 launched on executor multi with executor id 484 parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/6d5869b3-2738-4c2b-9624-c82fbabe332b_calibrate_1125542_5.stdout parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/6d5869b3-2738-4c2b-9624-c82fbabe332b_calibrate_1125542_5.stderr parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/c625f50a-6e21-4583-9c3a-71d130963aad_characterizeImage_1125542_5.stdout parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/c625f50a-6e21-4583-9c3a-71d130963aad_characterizeImage_1125542_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:27:16 proc_lsst.multi:146 [INFO] found job 16889827 in provider astro proc_lsst.multi INFO: found job 16889827 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 424 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38355a3d3240891dacc1b4ad30b9a394 with result from task 424 parsl.dataflow.dflow DEBUG: Task 424 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 425 has memoization hash ed21ef88428eba8414099e6403de71c5 parsl.dataflow.memoization INFO: Task 425 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f7bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 425 try 0 launched on executor multi with executor id 485 parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/00610bf6-80fb-43d3-af39-7fca115e8812_calibrate_1125543_46.stdout parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/00610bf6-80fb-43d3-af39-7fca115e8812_calibrate_1125543_46.stderr parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/ed3e276f-6608-43dd-82b7-20ae5e8b96b9_characterizeImage_1125543_46.stdout parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/ed3e276f-6608-43dd-82b7-20ae5e8b96b9_characterizeImage_1125543_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:27:18 proc_lsst.multi:146 [INFO] found job 16889828 in provider astro proc_lsst.multi INFO: found job 16889828 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 316 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2f88140a94ac27069f6de84236d02c0 with result from task 316 parsl.dataflow.dflow DEBUG: Task 316 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 317 has memoization hash e47fa7209613d19fce44800ada4b9003 parsl.dataflow.memoization INFO: Task 317 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f31c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 317 try 0 launched on executor multi with executor id 486 parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/49527edb-b3f9-4ef5-94b8-8ef0c3e56f9b_calibrate_1125543_37.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/49527edb-b3f9-4ef5-94b8-8ef0c3e56f9b_calibrate_1125543_37.stderr parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/b0ad3336-2037-46ae-9f77-d21e3f611026_characterizeImage_1125543_37.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/b0ad3336-2037-46ae-9f77-d21e3f611026_characterizeImage_1125543_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:27:19 proc_lsst.multi:146 [INFO] found job 16889829 in provider astro proc_lsst.multi INFO: found job 16889829 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 418 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e56ef9e565b53fef1af5e79b6527efcf with result from task 418 parsl.dataflow.dflow DEBUG: Task 418 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 419 has memoization hash 412ba548b076590134b7ff7b1f84627e parsl.dataflow.memoization INFO: Task 419 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f7b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 419 try 0 launched on executor multi with executor id 487 parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/3a983f77-ce8c-4606-9294-efd0481846cf_calibrate_1125542_3.stdout parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/3a983f77-ce8c-4606-9294-efd0481846cf_calibrate_1125542_3.stderr parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/7fb8c21b-e30b-4953-8e2d-90aec3561689_characterizeImage_1125542_3.stdout parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/7fb8c21b-e30b-4953-8e2d-90aec3561689_characterizeImage_1125542_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:27:20 proc_lsst.multi:146 [INFO] found job 16889830 in provider astro proc_lsst.multi INFO: found job 16889830 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:27:21 proc_lsst.multi:146 [INFO] found job 16889831 in provider astro proc_lsst.multi INFO: found job 16889831 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 286 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f89c092345a985ded52a980d6bf341f with result from task 286 parsl.dataflow.dflow DEBUG: Task 286 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 287 has memoization hash 1efd37cb7e953ae3e421a65fddfd5cf6 parsl.dataflow.memoization INFO: Task 287 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811effce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 287 try 0 launched on executor multi with executor id 488 parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/833b3417-9ae9-482e-8edb-211f94d8cb5d_calibrate_1125543_17.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/833b3417-9ae9-482e-8edb-211f94d8cb5d_calibrate_1125543_17.stderr parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/8f52e894-c89f-4ade-a261-ecaa4a38e820_characterizeImage_1125543_17.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/8f52e894-c89f-4ade-a261-ecaa4a38e820_characterizeImage_1125543_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:27:22 proc_lsst.multi:146 [INFO] found job 16889832 in provider astro proc_lsst.multi INFO: found job 16889832 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 442 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc7c53a25e34f673a1e4add49aa12bf3 with result from task 442 parsl.dataflow.dflow DEBUG: Task 442 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 443 has memoization hash 5826ee3b8570c633a345466e2a7d1746 parsl.dataflow.memoization INFO: Task 443 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f7ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 443 try 0 launched on executor multi with executor id 489 parsl.dataflow.dflow INFO: Standard output for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/5061ba44-d555-4590-a063-2fb27ef1276d_calibrate_1125541_45.stdout parsl.dataflow.dflow INFO: Standard error for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/5061ba44-d555-4590-a063-2fb27ef1276d_calibrate_1125541_45.stderr parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/3aabb4da-7ee4-4f82-8c99-490629014bcb_characterizeImage_1125541_45.stdout parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/3aabb4da-7ee4-4f82-8c99-490629014bcb_characterizeImage_1125541_45.stderr parsl.dataflow.dflow INFO: Task 346 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a12515cf4df68a64c3a910b1db0914f with result from task 346 parsl.dataflow.dflow DEBUG: Task 346 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 347 has memoization hash ede881e5d64097e6c73907f544d67d6c parsl.dataflow.memoization INFO: Task 347 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f33ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 347 try 0 launched on executor multi with executor id 490 parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/3fe7e129-3054-4290-b9eb-ff7a7df41c2e_calibrate_1125541_3.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/3fe7e129-3054-4290-b9eb-ff7a7df41c2e_calibrate_1125541_3.stderr parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/bccca5bb-37d3-46f0-bcab-d18ed8e2e682_characterizeImage_1125541_3.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/bccca5bb-37d3-46f0-bcab-d18ed8e2e682_characterizeImage_1125541_3.stderr parsl.dataflow.dflow INFO: Task 370 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6888aaa228551b4842b2c09c7ebbf56 with result from task 370 parsl.dataflow.dflow DEBUG: Task 370 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 371 has memoization hash d557ce68e0dce6d2c2c2a2d8817bb2f6 parsl.dataflow.memoization INFO: Task 371 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f794e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 371 try 0 launched on executor multi with executor id 491 parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/d22bb2e6-5f1d-4e52-8cb6-46d2fcda3bf4_calibrate_1125541_53.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/d22bb2e6-5f1d-4e52-8cb6-46d2fcda3bf4_calibrate_1125541_53.stderr parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/834014b0-d80b-493b-8b20-2b77742cfd2f_characterizeImage_1125541_53.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/834014b0-d80b-493b-8b20-2b77742cfd2f_characterizeImage_1125541_53.stderr parsl.dataflow.dflow INFO: Task 379 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69a5b7c8c1070b4151d2b0b839743fb9 with result from task 379 parsl.dataflow.dflow DEBUG: Task 379 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 380 has memoization hash 1fbc30d4f400d1d7983ec3d0b7f16d4e parsl.dataflow.memoization INFO: Task 380 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f79e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 380 try 0 launched on executor multi with executor id 492 parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/26488ece-6316-4f52-ac74-b4e52becc16b_calibrate_1125543_30.stdout parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/26488ece-6316-4f52-ac74-b4e52becc16b_calibrate_1125543_30.stderr parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/34b82464-82b0-4450-8b12-c4ca7a5e2ee7_characterizeImage_1125543_30.stdout parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/34b82464-82b0-4450-8b12-c4ca7a5e2ee7_characterizeImage_1125543_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:27:27 proc_lsst.multi:146 [INFO] found job 16889833 in provider astro proc_lsst.multi INFO: found job 16889833 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 367 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fdec887d2aac35275d3b075fc8ce9945 with result from task 367 parsl.dataflow.dflow DEBUG: Task 367 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 368 has memoization hash 0b2afa33f2fcc2a08d7d83f140e14d84 parsl.dataflow.memoization INFO: Task 368 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f791c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 368 try 0 launched on executor multi with executor id 493 parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/62e59b8c-92be-4122-804a-ccc12066cb6c_calibrate_1125543_5.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/62e59b8c-92be-4122-804a-ccc12066cb6c_calibrate_1125543_5.stderr parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/b1b96af1-ed22-4a53-8654-1f6010a5d895_characterizeImage_1125543_5.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/b1b96af1-ed22-4a53-8654-1f6010a5d895_characterizeImage_1125543_5.stderr parsl.dataflow.dflow INFO: Task 391 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c060318160e09e93f3a8583e151b0315 with result from task 391 parsl.dataflow.dflow DEBUG: Task 391 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 392 has memoization hash a4ba2b53aa4d32787075461c5c5c3576 parsl.dataflow.memoization INFO: Task 392 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f7aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 392 try 0 launched on executor multi with executor id 494 parsl.dataflow.dflow INFO: Standard output for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/04b39646-b7f5-4aab-8c05-adc9fb8f63b0_calibrate_1125543_23.stdout parsl.dataflow.dflow INFO: Standard error for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/04b39646-b7f5-4aab-8c05-adc9fb8f63b0_calibrate_1125543_23.stderr parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/9eb275eb-1045-439f-868c-8588a45b81ce_characterizeImage_1125543_23.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/9eb275eb-1045-439f-868c-8588a45b81ce_characterizeImage_1125543_23.stderr parsl.dataflow.dflow INFO: Task 109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f04ca6497f4da78c1f7e5816ac661d69 with result from task 109 parsl.dataflow.dflow DEBUG: Task 109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 110 has memoization hash 1282ff24ed036bcc838775838343243e parsl.dataflow.memoization INFO: Task 110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eac400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 495 parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/2566dd52-8d9e-42a9-9f47-fa2f4b4e93fd_calibrate_1125542_19.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/2566dd52-8d9e-42a9-9f47-fa2f4b4e93fd_calibrate_1125542_19.stderr parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/7b591715-5591-4cb2-ae0c-36726db40cec_characterizeImage_1125542_19.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/7b591715-5591-4cb2-ae0c-36726db40cec_characterizeImage_1125542_19.stderr parsl.dataflow.dflow INFO: Task 412 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64550ef3cefc01039ea952de1e79fe31 with result from task 412 parsl.dataflow.dflow DEBUG: Task 412 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 413 has memoization hash 301263a3a278f2dfe93f22c3aaeebf97 parsl.dataflow.memoization INFO: Task 413 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f7b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 413 try 0 launched on executor multi with executor id 496 parsl.dataflow.dflow INFO: Standard output for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/d93752b0-8519-41f8-93de-b89ab204bc6f_calibrate_1125541_39.stdout parsl.dataflow.dflow INFO: Standard error for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/d93752b0-8519-41f8-93de-b89ab204bc6f_calibrate_1125541_39.stderr parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/5c5674d0-e8b2-453e-ba3a-2cd57655d869_characterizeImage_1125541_39.stdout parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/5c5674d0-e8b2-453e-ba3a-2cd57655d869_characterizeImage_1125541_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:27:28 proc_lsst.multi:146 [INFO] found job 16889834 in provider astro proc_lsst.multi INFO: found job 16889834 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 334 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5055693394285aedeb36964006cce3d with result from task 334 parsl.dataflow.dflow DEBUG: Task 334 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 335 has memoization hash 55e42475bf3f10a38a3a704bea533f4c parsl.dataflow.memoization INFO: Task 335 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f32f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 335 try 0 launched on executor multi with executor id 497 parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/3ab515e3-cb5e-48e7-8f96-c65e93df04d2_calibrate_1125542_18.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/3ab515e3-cb5e-48e7-8f96-c65e93df04d2_calibrate_1125542_18.stderr parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/0ef28c78-8097-4565-b312-36f54e69391c_characterizeImage_1125542_18.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/0ef28c78-8097-4565-b312-36f54e69391c_characterizeImage_1125542_18.stderr parsl.dataflow.dflow INFO: Task 340 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ecff4f05f5b7402097dcb6392dda78eb with result from task 340 parsl.dataflow.dflow DEBUG: Task 340 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 341 has memoization hash 9c6781a2f67966abb278db45cc98e608 parsl.dataflow.memoization INFO: Task 341 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f33560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 341 try 0 launched on executor multi with executor id 498 parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/9c536bc0-a8ab-4d33-93d8-9d4a872ae3ca_calibrate_1125543_28.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/9c536bc0-a8ab-4d33-93d8-9d4a872ae3ca_calibrate_1125543_28.stderr parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/3d8e8ea3-f7a1-4f55-9417-564a6b799d5a_characterizeImage_1125543_28.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/3d8e8ea3-f7a1-4f55-9417-564a6b799d5a_characterizeImage_1125543_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:27:29 proc_lsst.multi:146 [INFO] found job 16889835 in provider astro proc_lsst.multi INFO: found job 16889835 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 373 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2e54ed358fd2348c77eaa49b9991b4e with result from task 373 parsl.dataflow.dflow DEBUG: Task 373 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 374 has memoization hash 297dc023ca432ff172c7f488114ccd5e parsl.dataflow.memoization INFO: Task 374 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f79800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 374 try 0 launched on executor multi with executor id 499 parsl.dataflow.dflow INFO: Standard output for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/28dc7b19-ae4a-47c0-a638-dd7e31bfa6d8_calibrate_1125542_39.stdout parsl.dataflow.dflow INFO: Standard error for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/28dc7b19-ae4a-47c0-a638-dd7e31bfa6d8_calibrate_1125542_39.stderr parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/7cb1c164-33f5-45e7-a4c7-70629b295801_characterizeImage_1125542_39.stdout parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/7cb1c164-33f5-45e7-a4c7-70629b295801_characterizeImage_1125542_39.stderr parsl.dataflow.dflow INFO: Task 430 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e6274f759f38c06f398425c15cb37be with result from task 430 parsl.dataflow.dflow DEBUG: Task 430 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 431 has memoization hash a030bcf8e07ae61f38cd4491cb05635a parsl.dataflow.memoization INFO: Task 431 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbc0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 431 try 0 launched on executor multi with executor id 500 parsl.dataflow.dflow INFO: Standard output for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/8ffd09ff-c100-48bd-8e03-75beed597f99_calibrate_1125541_23.stdout parsl.dataflow.dflow INFO: Standard error for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/8ffd09ff-c100-48bd-8e03-75beed597f99_calibrate_1125541_23.stderr parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/2c0df08f-9946-4340-bad2-a667913ea98d_characterizeImage_1125541_23.stdout parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/2c0df08f-9946-4340-bad2-a667913ea98d_characterizeImage_1125541_23.stderr parsl.dataflow.dflow INFO: Task 463 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b4f8f99ea0f7e6dceb8d02aa549da6a with result from task 463 parsl.dataflow.dflow DEBUG: Task 463 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 464 has memoization hash 2de0f5afc2beeb1e9c006046b1987782 parsl.dataflow.memoization INFO: Task 464 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfe2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 464 try 0 launched on executor multi with executor id 501 parsl.dataflow.dflow INFO: Standard output for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/5d5a93d5-3bbe-46f0-bff9-e57a6ee77e80_calibrate_1125543_21.stdout parsl.dataflow.dflow INFO: Standard error for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/5d5a93d5-3bbe-46f0-bff9-e57a6ee77e80_calibrate_1125543_21.stderr parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/77060bad-dfbf-498e-8b90-56db5c4b4019_characterizeImage_1125543_21.stdout parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/77060bad-dfbf-498e-8b90-56db5c4b4019_characterizeImage_1125543_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:27:30 proc_lsst.multi:146 [INFO] found job 16889836 in provider astro proc_lsst.multi INFO: found job 16889836 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 403 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37ac5b112cfa18deb3b681bcb77e00c9 with result from task 403 parsl.dataflow.dflow DEBUG: Task 403 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 404 has memoization hash 7d7f92741e28e556bc096b319d7c62f6 parsl.dataflow.memoization INFO: Task 404 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbd8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 404 try 0 launched on executor multi with executor id 502 parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/1a025eeb-9a55-49e9-bdd9-3ac3fed7f95f_calibrate_1125542_59.stdout parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/1a025eeb-9a55-49e9-bdd9-3ac3fed7f95f_calibrate_1125542_59.stderr parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/112b936d-6d78-432e-a208-07e90a433e52_characterizeImage_1125542_59.stdout parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/112b936d-6d78-432e-a208-07e90a433e52_characterizeImage_1125542_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:27:31 proc_lsst.multi:146 [INFO] found job 16889837 in provider astro proc_lsst.multi INFO: found job 16889837 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 472 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab33c25d6e487ec9495b3a6d02fd4291 with result from task 472 parsl.dataflow.dflow DEBUG: Task 472 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 473 has memoization hash 6aeefb232b2f2843b91fda495527299a parsl.dataflow.memoization INFO: Task 473 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbfce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 473 try 0 launched on executor multi with executor id 503 parsl.dataflow.dflow INFO: Standard output for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/2b0afa68-bc68-4100-8dd0-325d982f2224_calibrate_1125542_41.stdout parsl.dataflow.dflow INFO: Standard error for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/2b0afa68-bc68-4100-8dd0-325d982f2224_calibrate_1125542_41.stderr parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/283d78b9-dcab-4b02-be4b-411548279a01_characterizeImage_1125542_41.stdout parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/283d78b9-dcab-4b02-be4b-411548279a01_characterizeImage_1125542_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:27:32 proc_lsst.multi:146 [INFO] found job 16889838 in provider astro proc_lsst.multi INFO: found job 16889838 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 160 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3357fb811a4759e1120787bd2446a7f with result from task 160 parsl.dataflow.dflow DEBUG: Task 160 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 161 has memoization hash f1f60174ab10a33a617d5af76b9eddc9 parsl.dataflow.memoization INFO: Task 161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eaf920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 161 try 0 launched on executor multi with executor id 504 parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/ba24f6e0-f977-403f-9928-5b60c650e2a4_calibrate_1125542_61.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/ba24f6e0-f977-403f-9928-5b60c650e2a4_calibrate_1125542_61.stderr parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/909a6832-175b-405e-81b6-4faa248b5a28_characterizeImage_1125542_61.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/909a6832-175b-405e-81b6-4faa248b5a28_characterizeImage_1125542_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:27:33 proc_lsst.multi:146 [INFO] found job 16889839 in provider astro proc_lsst.multi INFO: found job 16889839 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 436 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 414f006eb60a2e1521638e867a636ecb with result from task 436 parsl.dataflow.dflow DEBUG: Task 436 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 437 has memoization hash 3d92e8ec53db1fad8fb019381e9a9f75 parsl.dataflow.memoization INFO: Task 437 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbc720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 437 try 0 launched on executor multi with executor id 505 parsl.dataflow.dflow INFO: Standard output for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/787c3c0e-0222-410a-8434-ce938699af47_calibrate_1125543_45.stdout parsl.dataflow.dflow INFO: Standard error for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/787c3c0e-0222-410a-8434-ce938699af47_calibrate_1125543_45.stderr parsl.dataflow.dflow INFO: Standard output for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/6746dff4-adb2-468f-b598-dc028d3a7d66_characterizeImage_1125543_45.stdout parsl.dataflow.dflow INFO: Standard error for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/6746dff4-adb2-468f-b598-dc028d3a7d66_characterizeImage_1125543_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:27:34 proc_lsst.multi:146 [INFO] found job 16889840 in provider astro proc_lsst.multi INFO: found job 16889840 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 388 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6615c80477cf5d80918311a7ba2113ee with result from task 388 parsl.dataflow.dflow DEBUG: Task 388 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 389 has memoization hash 74d6b35eb815e13690d225089fd481e4 parsl.dataflow.memoization INFO: Task 389 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f7a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 389 try 0 launched on executor multi with executor id 506 parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/9258693f-fb73-4ad7-9f52-e6f6ccdf7983_calibrate_1125541_47.stdout parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/9258693f-fb73-4ad7-9f52-e6f6ccdf7983_calibrate_1125541_47.stderr parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/799012a6-a874-4e9c-b41e-160c5dfbb2ea_characterizeImage_1125541_47.stdout parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/799012a6-a874-4e9c-b41e-160c5dfbb2ea_characterizeImage_1125541_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:27:35 proc_lsst.multi:146 [INFO] found job 16889841 in provider astro proc_lsst.multi INFO: found job 16889841 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:27:36 proc_lsst.multi:146 [INFO] found job 16889842 in provider astro proc_lsst.multi INFO: found job 16889842 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 28 try 1 failed parsl.dataflow.dflow ERROR: Task 28 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 4ba87d1e2e9a26e3da9bbfc1f5d0ed84 with result from task 28 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 29 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 29 try 0 failed parsl.dataflow.dflow INFO: Task 29 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/726a633f-17c5-48e5-8e5a-b0fe44c73317_calibrate_1125541_2.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/726a633f-17c5-48e5-8e5a-b0fe44c73317_calibrate_1125541_2.stderr parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/1fa5e94f-b9b3-4e9f-8222-a315fa75f121_characterizeImage_1125541_2.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/1fa5e94f-b9b3-4e9f-8222-a315fa75f121_characterizeImage_1125541_2.stderr parsl.dataflow.dflow INFO: Task 415 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d2b0440dd73dc4728d1d02402dd2dac with result from task 415 parsl.dataflow.dflow DEBUG: Task 415 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 416 has memoization hash c111e3b22e1cbf7595265ca89beecf2f parsl.dataflow.memoization INFO: Task 416 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f7b2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 416 try 0 launched on executor multi with executor id 507 parsl.dataflow.dflow INFO: Standard output for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/964ecb9d-8ddb-4554-acb1-0a918ae11a19_calibrate_1125542_55.stdout parsl.dataflow.dflow INFO: Standard error for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/964ecb9d-8ddb-4554-acb1-0a918ae11a19_calibrate_1125542_55.stderr parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/251a135e-d94e-4393-8da9-814b247bb8a8_characterizeImage_1125542_55.stdout parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/251a135e-d94e-4393-8da9-814b247bb8a8_characterizeImage_1125542_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:27:37 proc_lsst.multi:146 [INFO] found job 16889843 in provider astro proc_lsst.multi INFO: found job 16889843 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:27:38 proc_lsst.multi:146 [INFO] found job 16889844 in provider astro proc_lsst.multi INFO: found job 16889844 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 394 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77e35c5f0d9d802b51cd1abc9fe2a85a with result from task 394 parsl.dataflow.dflow DEBUG: Task 394 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 395 has memoization hash 2b874973087c6dead363d0571e103191 parsl.dataflow.memoization INFO: Task 395 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbe020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 395 try 0 launched on executor multi with executor id 508 parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/3cb6f405-61b6-42db-b91f-fed48166d2cc_calibrate_1125543_29.stdout parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/3cb6f405-61b6-42db-b91f-fed48166d2cc_calibrate_1125543_29.stderr parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/23f7349a-46cf-49a3-b08e-741c8a2bb8fe_characterizeImage_1125543_29.stdout parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/23f7349a-46cf-49a3-b08e-741c8a2bb8fe_characterizeImage_1125543_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:27:39 proc_lsst.multi:146 [INFO] found job 16889845 in provider astro proc_lsst.multi INFO: found job 16889845 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:27:40 proc_lsst.multi:146 [INFO] found job 16889846 in provider astro proc_lsst.multi INFO: found job 16889846 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:27:41 proc_lsst.multi:146 [INFO] found job 16889847 in provider astro proc_lsst.multi INFO: found job 16889847 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:27:42 proc_lsst.multi:146 [INFO] found job 16889848 in provider astro proc_lsst.multi INFO: found job 16889848 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:27:43 proc_lsst.multi:146 [INFO] found job 16889852 in provider astro proc_lsst.multi INFO: found job 16889852 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 307 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e26399a39d871f07f5b2bf02df8045d with result from task 307 parsl.dataflow.dflow DEBUG: Task 307 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 308 has memoization hash 1349c74f40eaac4f86b2a8134a055bfe parsl.dataflow.memoization INFO: Task 308 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f31300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 308 try 0 launched on executor multi with executor id 509 parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/592678ee-00b7-41a7-9077-a03f23eb14d5_calibrate_1125542_1.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/592678ee-00b7-41a7-9077-a03f23eb14d5_calibrate_1125542_1.stderr parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/ac706922-c293-4d4e-9e01-b6914ac04119_characterizeImage_1125542_1.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/ac706922-c293-4d4e-9e01-b6914ac04119_characterizeImage_1125542_1.stderr parsl.dataflow.dflow INFO: Task 226 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7315fe536273316a6b38cbbc7d3e22a1 with result from task 226 parsl.dataflow.dflow DEBUG: Task 226 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 227 has memoization hash 51ba899a2b91877170a9cc6835ac13f2 parsl.dataflow.memoization INFO: Task 227 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed7f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 227 try 0 launched on executor multi with executor id 510 parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/2839e1e5-f342-419d-97a0-4ff5a8f632e3_calibrate_1125543_51.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/2839e1e5-f342-419d-97a0-4ff5a8f632e3_calibrate_1125543_51.stderr parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/0dd4318d-571a-44d6-915e-e6ee04157ddf_characterizeImage_1125543_51.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/0dd4318d-571a-44d6-915e-e6ee04157ddf_characterizeImage_1125543_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:27:44 proc_lsst.multi:146 [INFO] found job 16889853 in provider astro proc_lsst.multi INFO: found job 16889853 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 427 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69acdeff407f14eb8744ced7ad1e1f46 with result from task 427 parsl.dataflow.dflow DEBUG: Task 427 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 428 has memoization hash 2a4a9abe62a1d966abec37dd7b282fc9 parsl.dataflow.memoization INFO: Task 428 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbe3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 428 try 0 launched on executor multi with executor id 511 parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/133fca33-f817-4e66-ad73-3140723d0224_calibrate_1125541_60.stdout parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/133fca33-f817-4e66-ad73-3140723d0224_calibrate_1125541_60.stderr parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/6ff603f2-6cd6-464e-b221-0b77dced7b8e_characterizeImage_1125541_60.stdout parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/6ff603f2-6cd6-464e-b221-0b77dced7b8e_characterizeImage_1125541_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:27:45 proc_lsst.multi:146 [INFO] found job 16889854 in provider astro proc_lsst.multi INFO: found job 16889854 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 484 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7b9987a1f23630a934b56b5d07e349f with result from task 484 parsl.dataflow.dflow DEBUG: Task 484 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 485 has memoization hash 109e37cce79b23cffb552b7b1dd0e23a parsl.dataflow.memoization INFO: Task 485 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfc720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 485 try 0 launched on executor multi with executor id 512 parsl.dataflow.dflow INFO: Standard output for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/307330e0-a2b3-4b96-aea4-7778e5a6a9fa_calibrate_1125543_53.stdout parsl.dataflow.dflow INFO: Standard error for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/307330e0-a2b3-4b96-aea4-7778e5a6a9fa_calibrate_1125543_53.stderr parsl.dataflow.dflow INFO: Standard output for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/bc1ee523-d07a-4040-b03f-aa79d12946f7_characterizeImage_1125543_53.stdout parsl.dataflow.dflow INFO: Standard error for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/bc1ee523-d07a-4040-b03f-aa79d12946f7_characterizeImage_1125543_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:27:46 proc_lsst.multi:146 [INFO] found job 16889855 in provider astro proc_lsst.multi INFO: found job 16889855 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:27:47 proc_lsst.multi:146 [INFO] found job 16889856 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889856 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:27:48 proc_lsst.multi:146 [INFO] found job 16889857 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889857 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.dataflow.dflow INFO: Task 433 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b14316c4be2567381868c5d146574c96 with result from task 433 parsl.dataflow.dflow DEBUG: Task 433 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 434 has memoization hash e84699c962f373190a07aff13ee7d26c parsl.dataflow.memoization INFO: Task 434 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbc400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 434 try 0 launched on executor multi with executor id 513 parsl.dataflow.dflow INFO: Standard output for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/df7d917f-84e4-44b6-beaa-e2c05b19e48a_calibrate_1125542_45.stdout parsl.dataflow.dflow INFO: Standard error for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/df7d917f-84e4-44b6-beaa-e2c05b19e48a_calibrate_1125542_45.stderr parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/7a64743c-eb25-4915-ad46-b65b1ad3fb80_characterizeImage_1125542_45.stdout parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/7a64743c-eb25-4915-ad46-b65b1ad3fb80_characterizeImage_1125542_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:27:49 proc_lsst.multi:146 [INFO] found job 16889858 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889858 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.dataflow.dflow INFO: Task 457 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c51666a053c3dc137d53932a3a19e00a with result from task 457 parsl.dataflow.dflow DEBUG: Task 457 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 458 has memoization hash 5983840e2833393494bc57aadd309b15 parsl.dataflow.memoization INFO: Task 458 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbf1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 458 try 0 launched on executor multi with executor id 514 parsl.dataflow.dflow INFO: Standard output for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/a295e6ed-4da2-4ef3-8a22-a19b68c7e206_calibrate_1125541_34.stdout parsl.dataflow.dflow INFO: Standard error for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/a295e6ed-4da2-4ef3-8a22-a19b68c7e206_calibrate_1125541_34.stderr parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/b02bc6cb-4693-47d2-967f-7f96ea189338_characterizeImage_1125541_34.stdout parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/b02bc6cb-4693-47d2-967f-7f96ea189338_characterizeImage_1125541_34.stderr parsl.dataflow.dflow INFO: Task 232 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21c9a1334b41cb4007c17f720561c14b with result from task 232 parsl.dataflow.dflow DEBUG: Task 232 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 233 has memoization hash 730055ce73ebd4197295599a86b24042 parsl.dataflow.memoization INFO: Task 233 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efc4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 233 try 0 launched on executor multi with executor id 515 parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/b967b554-56bd-4425-aef8-c03d68f5e294_calibrate_1125543_14.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/b967b554-56bd-4425-aef8-c03d68f5e294_calibrate_1125543_14.stderr parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/5bf29d16-6476-437b-9284-1826186ac188_characterizeImage_1125543_14.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/5bf29d16-6476-437b-9284-1826186ac188_characterizeImage_1125543_14.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 121 try 1 failed parsl.dataflow.dflow ERROR: Task 121 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry bb50db9f13537bf93733584b053700e1 with result from task 121 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 122 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 122 try 0 failed parsl.dataflow.dflow INFO: Task 122 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/f5c23209-ebc9-47a0-8167-0929bf434cd2_calibrate_1125542_2.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/f5c23209-ebc9-47a0-8167-0929bf434cd2_calibrate_1125542_2.stderr parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/4e8791e5-4edd-4a9d-ac14-e1b97579fc05_characterizeImage_1125542_2.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/4e8791e5-4edd-4a9d-ac14-e1b97579fc05_characterizeImage_1125542_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 22 try 1 failed parsl.dataflow.dflow ERROR: Task 22 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 70ca8e34191ccafee926a0ba43c4500c with result from task 22 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 23 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 23 try 0 failed parsl.dataflow.dflow INFO: Task 23 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/86f3ca01-246c-46af-afd3-3e518ea07a23_calibrate_1125543_2.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/86f3ca01-246c-46af-afd3-3e518ea07a23_calibrate_1125543_2.stderr parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/2000f40d-8eda-44b7-bbea-08ecd9dbb933_characterizeImage_1125543_2.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/2000f40d-8eda-44b7-bbea-08ecd9dbb933_characterizeImage_1125543_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:27:51 proc_lsst.multi:146 [INFO] found job 16889859 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889859 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:27:52 proc_lsst.multi:146 [INFO] found job 16889860 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889860 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:27:53 proc_lsst.multi:146 [INFO] found job 16889861 in provider escience_a40 proc_lsst.multi INFO: found job 16889861 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 409 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2bab56a15b404d37584bcf968a0653a2 with result from task 409 parsl.dataflow.dflow DEBUG: Task 409 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 410 has memoization hash 88bba1d7e90ac5241b244c6a2f354994 parsl.dataflow.memoization INFO: Task 410 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbd260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 410 try 0 launched on executor multi with executor id 516 parsl.dataflow.dflow INFO: Standard output for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/121ef870-c453-4464-acbf-9fdf96abc8f5_calibrate_1125543_1.stdout parsl.dataflow.dflow INFO: Standard error for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/121ef870-c453-4464-acbf-9fdf96abc8f5_calibrate_1125543_1.stderr parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/7bb1e95e-6059-46e9-85e9-5d958500a594_characterizeImage_1125543_1.stdout parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/7bb1e95e-6059-46e9-85e9-5d958500a594_characterizeImage_1125543_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:27:54 proc_lsst.multi:146 [INFO] found job 16889862 in provider escience_a40 proc_lsst.multi INFO: found job 16889862 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:27:55 proc_lsst.multi:146 [INFO] found job 16889863 in provider escience_a40 proc_lsst.multi INFO: found job 16889863 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 220 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb2d3774fe303951b8e6bbb743ef42e4 with result from task 220 parsl.dataflow.dflow DEBUG: Task 220 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 221 has memoization hash 0f6570d789a5dc41b198dac57958b44e parsl.dataflow.memoization INFO: Task 221 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed77e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 221 try 0 launched on executor multi with executor id 517 parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/1499a73c-ae0d-49c9-9593-844e30467897_calibrate_1125542_23.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/1499a73c-ae0d-49c9-9593-844e30467897_calibrate_1125542_23.stderr parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/169a6baa-89bc-435b-956e-f111c48c2894_characterizeImage_1125542_23.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/169a6baa-89bc-435b-956e-f111c48c2894_characterizeImage_1125542_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:27:56 proc_lsst.multi:146 [INFO] found job 16889864 in provider escience_a40 proc_lsst.multi INFO: found job 16889864 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:27:57 proc_lsst.multi:146 [INFO] found job 16889865 in provider escience_a40 proc_lsst.multi INFO: found job 16889865 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:27:58 proc_lsst.multi:146 [INFO] found job 16889866 in provider escience_a40 proc_lsst.multi INFO: found job 16889866 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 478 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f506f4331b80bfbbb2e6bfe759fddf2 with result from task 478 parsl.dataflow.dflow DEBUG: Task 478 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 479 has memoization hash 3b388c3d8557731e70f8347b8c0cd73d parsl.dataflow.memoization INFO: Task 479 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfc2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 479 try 0 launched on executor multi with executor id 518 parsl.dataflow.dflow INFO: Standard output for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/f7a38e55-4344-43b2-9e11-0a645e996697_calibrate_1125542_30.stdout parsl.dataflow.dflow INFO: Standard error for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/f7a38e55-4344-43b2-9e11-0a645e996697_calibrate_1125542_30.stderr parsl.dataflow.dflow INFO: Standard output for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/be1aa591-2dee-4679-bc36-579a930c5ee2_characterizeImage_1125542_30.stdout parsl.dataflow.dflow INFO: Standard error for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/be1aa591-2dee-4679-bc36-579a930c5ee2_characterizeImage_1125542_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:27:59 proc_lsst.multi:146 [INFO] found job 16889867 in provider escience_a40 proc_lsst.multi INFO: found job 16889867 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:00 proc_lsst.multi:146 [INFO] found job 16889868 in provider escience_a40 proc_lsst.multi INFO: found job 16889868 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 229 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91e188b88878c8f1a6886bae82ae04b6 with result from task 229 parsl.dataflow.dflow DEBUG: Task 229 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 230 has memoization hash 3f2a9983ab3290463434f91e88ec3f3d parsl.dataflow.memoization INFO: Task 230 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efc180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 230 try 0 launched on executor multi with executor id 519 parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/1fa7b20e-b7e9-44ee-94f1-c81caa1a7865_calibrate_1125542_12.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/1fa7b20e-b7e9-44ee-94f1-c81caa1a7865_calibrate_1125542_12.stderr parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/2cb660db-45e1-4d86-906d-48445abaa853_characterizeImage_1125542_12.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/2cb660db-45e1-4d86-906d-48445abaa853_characterizeImage_1125542_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:01 proc_lsst.multi:146 [INFO] found job 16889869 in provider escience_a40 proc_lsst.multi INFO: found job 16889869 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:02 proc_lsst.multi:146 [INFO] found job 16889870 in provider escience_a40 proc_lsst.multi INFO: found job 16889870 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:03 proc_lsst.multi:146 [INFO] found job 16889871 in provider escience_a40 proc_lsst.multi INFO: found job 16889871 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 183 parsl.dataflow.strategy DEBUG: Executor multi has 183 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22643663051856 2024-03-07 10:28:04 proc_lsst.multi:146 [INFO] found job 24403 in provider local proc_lsst.multi INFO: found job 24403 in provider local 2024-03-07 10:28:04 proc_lsst.multi:146 [INFO] found job 16889823 in provider astro proc_lsst.multi INFO: found job 16889823 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:05 proc_lsst.multi:146 [INFO] found job 16889824 in provider astro proc_lsst.multi INFO: found job 16889824 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 460 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 185eda72efae130db3545dc5d94bef0c with result from task 460 parsl.dataflow.dflow DEBUG: Task 460 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 461 has memoization hash 5f74818594d8b0aa29bef92b68c05a31 parsl.dataflow.memoization INFO: Task 461 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfe840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 461 try 0 launched on executor multi with executor id 520 parsl.dataflow.dflow INFO: Standard output for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/80f0e546-60fd-40d9-a601-877a5c8fd362_calibrate_1125543_47.stdout parsl.dataflow.dflow INFO: Standard error for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/80f0e546-60fd-40d9-a601-877a5c8fd362_calibrate_1125543_47.stderr parsl.dataflow.dflow INFO: Standard output for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/ec70f22b-743d-4797-8256-70b4268d8aef_characterizeImage_1125543_47.stdout parsl.dataflow.dflow INFO: Standard error for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/ec70f22b-743d-4797-8256-70b4268d8aef_characterizeImage_1125543_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:06 proc_lsst.multi:146 [INFO] found job 16889825 in provider astro proc_lsst.multi INFO: found job 16889825 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 469 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ce9806bf52116f7200be01b25914a06 with result from task 469 parsl.dataflow.dflow DEBUG: Task 469 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 470 has memoization hash a34200d2ed1aa4ccf9c0b73a86addd97 parsl.dataflow.memoization INFO: Task 470 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbf920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 470 try 0 launched on executor multi with executor id 521 parsl.dataflow.dflow INFO: Standard output for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/97649eeb-cfc5-493c-89c7-c206c5f8e1b1_calibrate_1125541_25.stdout parsl.dataflow.dflow INFO: Standard error for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/97649eeb-cfc5-493c-89c7-c206c5f8e1b1_calibrate_1125541_25.stderr parsl.dataflow.dflow INFO: Standard output for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/b9901117-6bed-4880-b228-7e75e54bd27f_characterizeImage_1125541_25.stdout parsl.dataflow.dflow INFO: Standard error for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/b9901117-6bed-4880-b228-7e75e54bd27f_characterizeImage_1125541_25.stderr parsl.dataflow.dflow INFO: Task 475 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ff76c3ae87bee64a6eca41e43bd8917 with result from task 475 parsl.dataflow.dflow DEBUG: Task 475 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 476 has memoization hash 8666d5f99a4009eab6ed7bd02213e69d parsl.dataflow.memoization INFO: Task 476 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfdee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 476 try 0 launched on executor multi with executor id 522 parsl.dataflow.dflow INFO: Standard output for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/9383e3d5-afff-4c79-90d2-701188178099_calibrate_1125543_60.stdout parsl.dataflow.dflow INFO: Standard error for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/9383e3d5-afff-4c79-90d2-701188178099_calibrate_1125543_60.stderr parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/24ac9036-f36a-4161-9add-f8bf82a2c2d5_characterizeImage_1125543_60.stdout parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/24ac9036-f36a-4161-9add-f8bf82a2c2d5_characterizeImage_1125543_60.stderr parsl.dataflow.dflow INFO: Task 454 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08d5c88cff836d533e437aa4c04eba68 with result from task 454 parsl.dataflow.dflow DEBUG: Task 454 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 455 has memoization hash 330d333c2ad198b43b45b08f72f682cd parsl.dataflow.memoization INFO: Task 455 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 455 try 0 launched on executor multi with executor id 523 parsl.dataflow.dflow INFO: Standard output for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/172646fa-d961-4879-90d0-3e5beec93c5f_calibrate_1125541_18.stdout parsl.dataflow.dflow INFO: Standard error for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/172646fa-d961-4879-90d0-3e5beec93c5f_calibrate_1125541_18.stderr parsl.dataflow.dflow INFO: Standard output for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/f253d20e-116a-431d-bca3-c385f0581b8f_characterizeImage_1125541_18.stdout parsl.dataflow.dflow INFO: Standard error for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/f253d20e-116a-431d-bca3-c385f0581b8f_characterizeImage_1125541_18.stderr parsl.dataflow.dflow INFO: Task 481 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 048d9cd72aa2f5216a7c93b0c4e8c647 with result from task 481 parsl.dataflow.dflow DEBUG: Task 481 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 482 has memoization hash 50a275fac94897b44869ca953ef1155c parsl.dataflow.memoization INFO: Task 482 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfc400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 482 try 0 launched on executor multi with executor id 524 parsl.dataflow.dflow INFO: Standard output for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/cc7250ee-4d88-44d2-ae8e-4d3be8d82ff1_calibrate_1125541_58.stdout parsl.dataflow.dflow INFO: Standard error for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/cc7250ee-4d88-44d2-ae8e-4d3be8d82ff1_calibrate_1125541_58.stderr parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/22d9dfc3-d37c-48e0-9c6e-781bf5b939ad_characterizeImage_1125541_58.stdout parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/22d9dfc3-d37c-48e0-9c6e-781bf5b939ad_characterizeImage_1125541_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:07 proc_lsst.multi:146 [INFO] found job 16889826 in provider astro proc_lsst.multi INFO: found job 16889826 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:09 proc_lsst.multi:146 [INFO] found job 16889827 in provider astro proc_lsst.multi INFO: found job 16889827 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 448 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9526a5530710980671cb15d7070814d0 with result from task 448 parsl.dataflow.dflow DEBUG: Task 448 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 449 has memoization hash dbd1f72b40b2d7cd165ea71737913508 parsl.dataflow.memoization INFO: Task 449 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbe840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 449 try 0 launched on executor multi with executor id 525 parsl.dataflow.dflow INFO: Standard output for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/d0eecf8a-8386-4d9d-997a-fb622f91d4f5_calibrate_1125542_24.stdout parsl.dataflow.dflow INFO: Standard error for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/d0eecf8a-8386-4d9d-997a-fb622f91d4f5_calibrate_1125542_24.stderr parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/12fba411-39ec-49d6-bd9a-1f1a444c81fa_characterizeImage_1125542_24.stdout parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/12fba411-39ec-49d6-bd9a-1f1a444c81fa_characterizeImage_1125542_24.stderr parsl.dataflow.dflow INFO: Task 526 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de3e696d23043e3c87118c283531c444 with result from task 526 parsl.dataflow.dflow DEBUG: Task 526 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 527 has memoization hash 7452604f01c6273d891ec59982cb15b1 parsl.dataflow.memoization INFO: Task 527 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dffba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 527 try 0 launched on executor multi with executor id 526 parsl.dataflow.dflow INFO: Standard output for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/6b53a997-35e5-4955-a904-f122bac7d88e_calibrate_1125541_5.stdout parsl.dataflow.dflow INFO: Standard error for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/6b53a997-35e5-4955-a904-f122bac7d88e_calibrate_1125541_5.stderr parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/ac4839a9-00a2-4b39-892d-1eb1c9701bef_characterizeImage_1125541_5.stdout parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/ac4839a9-00a2-4b39-892d-1eb1c9701bef_characterizeImage_1125541_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:10 proc_lsst.multi:146 [INFO] found job 16889828 in provider astro proc_lsst.multi INFO: found job 16889828 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 496 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4da8cb6ca8381f7254a105f364174d25 with result from task 496 parsl.dataflow.dflow DEBUG: Task 496 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 497 has memoization hash 5745b55cb3d90568f2860fb449fc95e0 parsl.dataflow.memoization INFO: Task 497 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfd300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 497 try 0 launched on executor multi with executor id 527 parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/afb2b54b-1c7a-4f11-b4c5-c1ae4f1ad023_calibrate_1125541_38.stdout parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/afb2b54b-1c7a-4f11-b4c5-c1ae4f1ad023_calibrate_1125541_38.stderr parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/fd430661-7ecc-41f5-bc0c-71394e6cb18d_characterizeImage_1125541_38.stdout parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/fd430661-7ecc-41f5-bc0c-71394e6cb18d_characterizeImage_1125541_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:10 proc_lsst.multi:146 [INFO] found job 16889829 in provider astro proc_lsst.multi INFO: found job 16889829 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 529 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de5a0c7c9879b337ca87238b7355053f with result from task 529 parsl.dataflow.dflow DEBUG: Task 529 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 530 has memoization hash b193802470311522e91f39fae7f0ed91 parsl.dataflow.memoization INFO: Task 530 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dffe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 530 try 0 launched on executor multi with executor id 528 parsl.dataflow.dflow INFO: Standard output for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/1ab26cb9-bca3-4569-930e-83a8295ac107_calibrate_1125542_22.stdout parsl.dataflow.dflow INFO: Standard error for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/1ab26cb9-bca3-4569-930e-83a8295ac107_calibrate_1125542_22.stderr parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/f38117a3-bf39-451a-b55d-4c3679815476_characterizeImage_1125542_22.stdout parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/f38117a3-bf39-451a-b55d-4c3679815476_characterizeImage_1125542_22.stderr parsl.dataflow.dflow INFO: Task 235 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e9abab7cb64f439e7eb9b78ef03ffc5f with result from task 235 parsl.dataflow.dflow DEBUG: Task 235 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 236 has memoization hash 5be368f95e430a4e03161888121f3113 parsl.dataflow.memoization INFO: Task 236 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efc7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 236 try 0 launched on executor multi with executor id 529 parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/b7abbf89-a282-4ec2-af58-625b681c2675_calibrate_1125543_34.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/b7abbf89-a282-4ec2-af58-625b681c2675_calibrate_1125543_34.stderr parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/7b03af8e-7bbb-447c-aefa-29a65bd50a6c_characterizeImage_1125543_34.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/7b03af8e-7bbb-447c-aefa-29a65bd50a6c_characterizeImage_1125543_34.stderr parsl.dataflow.dflow INFO: Task 238 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d2635cc09059694628ffd5d2ab3ad4b with result from task 238 parsl.dataflow.dflow DEBUG: Task 238 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 239 has memoization hash 13fcda6c490c071363e48a2d27346353 parsl.dataflow.memoization INFO: Task 239 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811efcae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 239 try 0 launched on executor multi with executor id 530 parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/9eed6213-652c-4da2-b451-13e113ef7331_calibrate_1125543_4.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/9eed6213-652c-4da2-b451-13e113ef7331_calibrate_1125543_4.stderr parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/259f35ef-f9c9-437c-a4bb-000d2b793a69_characterizeImage_1125543_4.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/259f35ef-f9c9-437c-a4bb-000d2b793a69_characterizeImage_1125543_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:12 proc_lsst.multi:146 [INFO] found job 16889830 in provider astro proc_lsst.multi INFO: found job 16889830 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 451 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08296ae5e544f5351cf6356d35f5fff6 with result from task 451 parsl.dataflow.dflow DEBUG: Task 451 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 452 has memoization hash 1ef8e289f3c878fc88acc7af0726258a parsl.dataflow.memoization INFO: Task 452 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbeb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 452 try 0 launched on executor multi with executor id 531 parsl.dataflow.dflow INFO: Standard output for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/c1b949d5-d4dc-4aa8-bfef-2ec05ba219d7_calibrate_1125541_15.stdout parsl.dataflow.dflow INFO: Standard error for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/c1b949d5-d4dc-4aa8-bfef-2ec05ba219d7_calibrate_1125541_15.stderr parsl.dataflow.dflow INFO: Standard output for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/aa571e3e-cdd3-455e-aa11-e5916dce620e_characterizeImage_1125541_15.stdout parsl.dataflow.dflow INFO: Standard error for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/aa571e3e-cdd3-455e-aa11-e5916dce620e_characterizeImage_1125541_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:13 proc_lsst.multi:146 [INFO] found job 16889831 in provider astro proc_lsst.multi INFO: found job 16889831 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:14 proc_lsst.multi:146 [INFO] found job 16889832 in provider astro proc_lsst.multi INFO: found job 16889832 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 532 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aaf1be6b87663ac5a46a89f49657ccfa with result from task 532 parsl.dataflow.dflow DEBUG: Task 532 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 533 has memoization hash 5a616c21e9f03f772c829ac1fa2001b2 parsl.dataflow.memoization INFO: Task 533 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811e54220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 533 try 0 launched on executor multi with executor id 532 parsl.dataflow.dflow INFO: Standard output for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/edb6de06-6730-43cf-82d7-0302cf0bbf03_calibrate_1125543_16.stdout parsl.dataflow.dflow INFO: Standard error for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/edb6de06-6730-43cf-82d7-0302cf0bbf03_calibrate_1125543_16.stderr parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/668a6fd8-81df-41a0-937e-f6bd18bbe983_characterizeImage_1125543_16.stdout parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/668a6fd8-81df-41a0-937e-f6bd18bbe983_characterizeImage_1125543_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:15 proc_lsst.multi:146 [INFO] found job 16889833 in provider astro proc_lsst.multi INFO: found job 16889833 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 487 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96677fbce79003546f30bab9145e72c0 with result from task 487 parsl.dataflow.dflow DEBUG: Task 487 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 488 has memoization hash cb6f3c4c7824d4d537293574b59c9dc1 parsl.dataflow.memoization INFO: Task 488 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 488 try 0 launched on executor multi with executor id 533 parsl.dataflow.dflow INFO: Standard output for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/3da2371b-6131-428c-a8a1-b0c5ccbb22bd_calibrate_1125542_44.stdout parsl.dataflow.dflow INFO: Standard error for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/3da2371b-6131-428c-a8a1-b0c5ccbb22bd_calibrate_1125542_44.stderr parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/bc4cc82c-0175-43e9-828b-370efa7736e1_characterizeImage_1125542_44.stdout parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/bc4cc82c-0175-43e9-828b-370efa7736e1_characterizeImage_1125542_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:16 proc_lsst.multi:146 [INFO] found job 16889834 in provider astro proc_lsst.multi INFO: found job 16889834 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:17 proc_lsst.multi:146 [INFO] found job 16889835 in provider astro proc_lsst.multi INFO: found job 16889835 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 499 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5887bd43c3964853c1014dd693b98bb with result from task 499 parsl.dataflow.dflow DEBUG: Task 499 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 500 has memoization hash 8044736169904b069581cea433eace31 parsl.dataflow.memoization INFO: Task 500 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfde40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 500 try 0 launched on executor multi with executor id 534 parsl.dataflow.dflow INFO: Standard output for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/4a4a81aa-6011-458c-8026-bce73cac2d55_calibrate_1125542_27.stdout parsl.dataflow.dflow INFO: Standard error for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/4a4a81aa-6011-458c-8026-bce73cac2d55_calibrate_1125542_27.stderr parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/e9012e41-2ee4-4e39-88ae-fc47786c5922_characterizeImage_1125542_27.stdout parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/e9012e41-2ee4-4e39-88ae-fc47786c5922_characterizeImage_1125542_27.stderr parsl.dataflow.dflow INFO: Task 520 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry baaa763f25a6830e71de5396b2ef7ed7 with result from task 520 parsl.dataflow.dflow DEBUG: Task 520 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 521 has memoization hash 112bb18bd448f11ca4f24477e7e43678 parsl.dataflow.memoization INFO: Task 521 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dff560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 521 try 0 launched on executor multi with executor id 535 parsl.dataflow.dflow INFO: Standard output for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/1562f373-1270-4d75-980f-042b522c70b8_calibrate_1125542_21.stdout parsl.dataflow.dflow INFO: Standard error for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/1562f373-1270-4d75-980f-042b522c70b8_calibrate_1125542_21.stderr parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/b33eae3d-2bfb-4ecf-80c5-d5ce9c8e2efa_characterizeImage_1125542_21.stdout parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/b33eae3d-2bfb-4ecf-80c5-d5ce9c8e2efa_characterizeImage_1125542_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:18 proc_lsst.multi:146 [INFO] found job 16889836 in provider astro proc_lsst.multi INFO: found job 16889836 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 445 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83d976af8638af01445f585550003e77 with result from task 445 parsl.dataflow.dflow DEBUG: Task 445 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 446 has memoization hash 0a077b7327ee01550a53aa8acb41e645 parsl.dataflow.memoization INFO: Task 446 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 446 try 0 launched on executor multi with executor id 536 parsl.dataflow.dflow INFO: Standard output for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/080c34f8-795b-488c-aac6-dc0b278499ff_calibrate_1125541_40.stdout parsl.dataflow.dflow INFO: Standard error for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/080c34f8-795b-488c-aac6-dc0b278499ff_calibrate_1125541_40.stderr parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/8deb6b88-a428-47b5-a5e0-da6f787cc2b7_characterizeImage_1125541_40.stdout parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/8deb6b88-a428-47b5-a5e0-da6f787cc2b7_characterizeImage_1125541_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:19 proc_lsst.multi:146 [INFO] found job 16889837 in provider astro proc_lsst.multi INFO: found job 16889837 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:20 proc_lsst.multi:146 [INFO] found job 16889838 in provider astro proc_lsst.multi INFO: found job 16889838 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:21 proc_lsst.multi:146 [INFO] found job 16889839 in provider astro proc_lsst.multi INFO: found job 16889839 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:22 proc_lsst.multi:146 [INFO] found job 16889840 in provider astro proc_lsst.multi INFO: found job 16889840 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 514 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c37b72b5a7773da0d49a66c60c9f5922 with result from task 514 parsl.dataflow.dflow DEBUG: Task 514 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 515 has memoization hash cfbbbd05e3762c6a64691f2ca1795a14 parsl.dataflow.memoization INFO: Task 515 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 515 try 0 launched on executor multi with executor id 537 parsl.dataflow.dflow INFO: Standard output for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/9f8916fb-0886-4f28-b808-49f5e34922fc_calibrate_1125542_54.stdout parsl.dataflow.dflow INFO: Standard error for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/9f8916fb-0886-4f28-b808-49f5e34922fc_calibrate_1125542_54.stderr parsl.dataflow.dflow INFO: Standard output for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/50b3a5b4-3d90-43ad-be10-9d269c2929af_characterizeImage_1125542_54.stdout parsl.dataflow.dflow INFO: Standard error for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/50b3a5b4-3d90-43ad-be10-9d269c2929af_characterizeImage_1125542_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:24 proc_lsst.multi:146 [INFO] found job 16889841 in provider astro proc_lsst.multi INFO: found job 16889841 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 502 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e52d39c8299a297d686bde9339e6ed9 with result from task 502 parsl.dataflow.dflow DEBUG: Task 502 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 503 has memoization hash 25723c069fb7bbaa4624e5f840520317 parsl.dataflow.memoization INFO: Task 503 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfdb20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 503 try 0 launched on executor multi with executor id 538 parsl.dataflow.dflow INFO: Standard output for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/695636d6-9c2e-4635-8e89-c70fcabb136c_calibrate_1125541_8.stdout parsl.dataflow.dflow INFO: Standard error for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/695636d6-9c2e-4635-8e89-c70fcabb136c_calibrate_1125541_8.stderr parsl.dataflow.dflow INFO: Standard output for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/a3826850-bb6b-4bb9-8d90-cd7eee67f99c_characterizeImage_1125541_8.stdout parsl.dataflow.dflow INFO: Standard error for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/a3826850-bb6b-4bb9-8d90-cd7eee67f99c_characterizeImage_1125541_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:25 proc_lsst.multi:146 [INFO] found job 16889842 in provider astro proc_lsst.multi INFO: found job 16889842 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:26 proc_lsst.multi:146 [INFO] found job 16889843 in provider astro proc_lsst.multi INFO: found job 16889843 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a72348daee8ac6e4cd0b201e7761b288 with result from task 101 parsl.dataflow.dflow DEBUG: Task 101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/d376594e-f605-4102-9a56-98b29324a468_calibrate_1125541_6.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/d376594e-f605-4102-9a56-98b29324a468_calibrate_1125541_6.stderr parsl.dataflow.dflow INFO: Task 508 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c709ff72f89e1e0550524bd8d378f4a with result from task 508 parsl.dataflow.dflow DEBUG: Task 508 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 509 has memoization hash 3cbbe4974867b8aa101ef7a650a710c5 parsl.dataflow.memoization INFO: Task 509 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfd440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 509 try 0 launched on executor multi with executor id 539 parsl.dataflow.dflow INFO: Standard output for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/a8c15e45-a640-497c-9dbf-c225b268a818_calibrate_1125542_46.stdout parsl.dataflow.dflow INFO: Standard error for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/a8c15e45-a640-497c-9dbf-c225b268a818_calibrate_1125542_46.stderr parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/69e93b2a-6c40-4982-8ddc-95550993a199_characterizeImage_1125542_46.stdout parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/69e93b2a-6c40-4982-8ddc-95550993a199_characterizeImage_1125542_46.stderr parsl.dataflow.dflow INFO: Task 493 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0ec7f722319fdf3eb3073cf40208f40 with result from task 493 parsl.dataflow.dflow DEBUG: Task 493 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 494 has memoization hash 5fa82d2c70c30fc3f4c885a52843cf76 parsl.dataflow.memoization INFO: Task 494 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 494 try 0 launched on executor multi with executor id 540 parsl.dataflow.dflow INFO: Standard output for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/4b9ec996-b49c-44f8-a030-261594729504_calibrate_1125543_40.stdout parsl.dataflow.dflow INFO: Standard error for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/4b9ec996-b49c-44f8-a030-261594729504_calibrate_1125543_40.stderr parsl.dataflow.dflow INFO: Standard output for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/504cde24-a2ac-4b31-b69e-831aed9871ea_characterizeImage_1125543_40.stdout parsl.dataflow.dflow INFO: Standard error for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/504cde24-a2ac-4b31-b69e-831aed9871ea_characterizeImage_1125543_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:27 proc_lsst.multi:146 [INFO] found job 16889844 in provider astro proc_lsst.multi INFO: found job 16889844 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:28 proc_lsst.multi:146 [INFO] found job 16889845 in provider astro proc_lsst.multi INFO: found job 16889845 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:30 proc_lsst.multi:146 [INFO] found job 16889846 in provider astro proc_lsst.multi INFO: found job 16889846 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 556 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a48ee9c60beb1a97d2355e3d4a2802d with result from task 556 parsl.dataflow.dflow DEBUG: Task 556 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 557 has memoization hash e8d5b17ddc4c0d5963dbf8b654b7e731 parsl.dataflow.memoization INFO: Task 557 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811e55940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 557 try 0 launched on executor multi with executor id 541 parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/da9481ab-a802-4695-9060-a0077bea28aa_calibrate_1125543_52.stdout parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/da9481ab-a802-4695-9060-a0077bea28aa_calibrate_1125543_52.stderr parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/ae187e77-39e8-4e18-be49-3d75fc85361c_characterizeImage_1125543_52.stdout parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/ae187e77-39e8-4e18-be49-3d75fc85361c_characterizeImage_1125543_52.stderr parsl.dataflow.dflow INFO: Task 544 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b105d92feacb79496075f47c0547aafe with result from task 544 parsl.dataflow.dflow DEBUG: Task 544 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 545 has memoization hash 41e4215446517498e70871a0ce28f6b7 parsl.dataflow.memoization INFO: Task 545 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811e54e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:31 proc_lsst.multi:146 [INFO] found job 16889847 in provider astro proc_lsst.multi INFO: found job 16889847 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Parsl task 545 try 0 launched on executor multi with executor id 542 parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/86213a17-a98c-409d-81b8-98c344f70438_calibrate_1125541_50.stdout parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/86213a17-a98c-409d-81b8-98c344f70438_calibrate_1125541_50.stderr parsl.dataflow.dflow INFO: Standard output for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/5a1b7098-e4c5-48d1-a753-70a565811ac7_characterizeImage_1125541_50.stdout parsl.dataflow.dflow INFO: Standard error for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/5a1b7098-e4c5-48d1-a753-70a565811ac7_characterizeImage_1125541_50.stderr parsl.dataflow.dflow INFO: Task 490 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4fc8826e0a66c157fa587f5971ae1f2 with result from task 490 parsl.dataflow.dflow DEBUG: Task 490 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 491 has memoization hash 58d8ddd85bd84742aa42fdc637c06011 parsl.dataflow.memoization INFO: Task 491 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfcd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 491 try 0 launched on executor multi with executor id 543 parsl.dataflow.dflow INFO: Standard output for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/8cf6fe2e-6185-43ba-81fd-640004e3db4e_calibrate_1125542_14.stdout parsl.dataflow.dflow INFO: Standard error for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/8cf6fe2e-6185-43ba-81fd-640004e3db4e_calibrate_1125542_14.stderr parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/9398b1f1-6694-4d31-a65b-69a4ac8ed3fe_characterizeImage_1125542_14.stdout parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/9398b1f1-6694-4d31-a65b-69a4ac8ed3fe_characterizeImage_1125542_14.stderr parsl.dataflow.dflow INFO: Task 541 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a602cf2779dbb4be7d94c1191257a89 with result from task 541 parsl.dataflow.dflow DEBUG: Task 541 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 542 has memoization hash 5428772dda4759202606a021c581045c parsl.dataflow.memoization INFO: Task 542 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbf100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 542 try 0 launched on executor multi with executor id 544 parsl.dataflow.dflow INFO: Standard output for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/f306719e-b0b2-4804-b2cf-2812b0e4210d_calibrate_1125542_56.stdout parsl.dataflow.dflow INFO: Standard error for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/f306719e-b0b2-4804-b2cf-2812b0e4210d_calibrate_1125542_56.stderr parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/ea7a1c38-9e01-4085-99ed-2ee303a16fc9_characterizeImage_1125542_56.stdout parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/ea7a1c38-9e01-4085-99ed-2ee303a16fc9_characterizeImage_1125542_56.stderr parsl.dataflow.dflow INFO: Task 538 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09d36cf939d6769b0040f1daa01e6e0d with result from task 538 parsl.dataflow.dflow DEBUG: Task 538 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 539 has memoization hash 219608d91b3757f0e1a87959f3c5447e parsl.dataflow.memoization INFO: Task 539 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811e54860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:34 proc_lsst.multi:146 [INFO] found job 16889848 in provider astro proc_lsst.multi INFO: found job 16889848 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Parsl task 539 try 0 launched on executor multi with executor id 545 parsl.dataflow.dflow INFO: Standard output for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/c0151ac4-1623-49f7-b930-964b53c76b22_calibrate_1125542_33.stdout parsl.dataflow.dflow INFO: Standard error for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/c0151ac4-1623-49f7-b930-964b53c76b22_calibrate_1125542_33.stderr parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/6c5df536-70a6-4fdf-820a-6abf12f53449_characterizeImage_1125542_33.stdout parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/6c5df536-70a6-4fdf-820a-6abf12f53449_characterizeImage_1125542_33.stderr parsl.dataflow.dflow INFO: Task 547 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7361d2a176c30636e6f86659cb65109b with result from task 547 parsl.dataflow.dflow DEBUG: Task 547 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 548 has memoization hash 7d9d49bdcb67cb5f3fde97d4e27daff7 parsl.dataflow.memoization INFO: Task 548 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811e54fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 548 try 0 launched on executor multi with executor id 546 parsl.dataflow.dflow INFO: Standard output for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/5002688a-bf9a-4c46-850b-7cd287de3b12_calibrate_1125541_11.stdout parsl.dataflow.dflow INFO: Standard error for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/5002688a-bf9a-4c46-850b-7cd287de3b12_calibrate_1125541_11.stderr parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/23e11865-02d8-4f48-afe4-7d613da8a790_characterizeImage_1125541_11.stdout parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/23e11865-02d8-4f48-afe4-7d613da8a790_characterizeImage_1125541_11.stderr parsl.dataflow.dflow INFO: Task 517 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b882c82f336c8447811a3c97d8229e83 with result from task 517 parsl.dataflow.dflow DEBUG: Task 517 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 518 has memoization hash 42435c7d72036ae416fa890f16ef34c8 parsl.dataflow.memoization INFO: Task 518 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dff240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 518 try 0 launched on executor multi with executor id 547 parsl.dataflow.dflow INFO: Standard output for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/65f9b44b-689f-4d7e-a90b-18dac112d2c5_calibrate_1125541_13.stdout parsl.dataflow.dflow INFO: Standard error for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/65f9b44b-689f-4d7e-a90b-18dac112d2c5_calibrate_1125541_13.stderr parsl.dataflow.dflow INFO: Standard output for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/51730513-f65f-448f-8804-f6e79d3cb228_characterizeImage_1125541_13.stdout parsl.dataflow.dflow INFO: Standard error for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/51730513-f65f-448f-8804-f6e79d3cb228_characterizeImage_1125541_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:36 proc_lsst.multi:146 [INFO] found job 16889852 in provider astro proc_lsst.multi INFO: found job 16889852 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 523 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1b3c55bf24bd1ef283947cdcb22bbc2 with result from task 523 parsl.dataflow.dflow DEBUG: Task 523 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 524 has memoization hash 3f45835d1b6caf58e966d6c6e130501d parsl.dataflow.memoization INFO: Task 524 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dff880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 524 try 0 launched on executor multi with executor id 548 parsl.dataflow.dflow INFO: Standard output for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/275e90cd-7d33-4e6d-9913-dccf02ef1811_calibrate_1125541_52.stdout parsl.dataflow.dflow INFO: Standard error for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/275e90cd-7d33-4e6d-9913-dccf02ef1811_calibrate_1125541_52.stderr parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/cc979036-214f-4d4d-a941-3b7629ea62f5_characterizeImage_1125541_52.stdout parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/cc979036-214f-4d4d-a941-3b7629ea62f5_characterizeImage_1125541_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:37 proc_lsst.multi:146 [INFO] found job 16889853 in provider astro proc_lsst.multi INFO: found job 16889853 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 26 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea497160b6500b6fbff6c8ec12c90eeb with result from task 26 parsl.dataflow.dflow DEBUG: Task 26 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/be26deb9-1cca-432b-822a-6a6f41138fa2_calibrate_1125541_26.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/be26deb9-1cca-432b-822a-6a6f41138fa2_calibrate_1125541_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:37 proc_lsst.multi:146 [INFO] found job 16889854 in provider astro proc_lsst.multi INFO: found job 16889854 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 53 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c9789f14cc8d65ae6792e4dae48cf2d with result from task 53 parsl.dataflow.dflow DEBUG: Task 53 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/adf89e09-fb66-4896-b320-94334b35eeae_calibrate_1125541_4.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/adf89e09-fb66-4896-b320-94334b35eeae_calibrate_1125541_4.stderr parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry efd943edd2990c44034c182934f99a5f with result from task 14 parsl.dataflow.dflow DEBUG: Task 14 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/763b996f-91a3-4df3-845d-2dd87bff2361_calibrate_1125542_58.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/763b996f-91a3-4df3-845d-2dd87bff2361_calibrate_1125542_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:38 proc_lsst.multi:146 [INFO] found job 16889855 in provider astro proc_lsst.multi INFO: found job 16889855 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4baf95301d3815441b7425351838dc79 with result from task 11 parsl.dataflow.dflow DEBUG: Task 11 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/0e5bf43c-a446-4ee3-806a-402d7664fbfd_calibrate_1125541_29.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/0e5bf43c-a446-4ee3-806a-402d7664fbfd_calibrate_1125541_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:39 proc_lsst.multi:146 [INFO] found job 16889856 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889856 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:41 proc_lsst.multi:146 [INFO] found job 16889857 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889857 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e2fe06ebf6d6afd19473ee017f56023 with result from task 17 parsl.dataflow.dflow DEBUG: Task 17 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/bedbcfe1-3275-4d33-b0d3-d60632191ee9_calibrate_1125542_60.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/bedbcfe1-3275-4d33-b0d3-d60632191ee9_calibrate_1125542_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:42 proc_lsst.multi:146 [INFO] found job 16889858 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889858 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.dataflow.dflow INFO: Task 149 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c466fed24cec5e5482e52d068fa1adaa with result from task 149 parsl.dataflow.dflow DEBUG: Task 149 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/cf79c1d8-1456-41d8-a784-34fd6545652a_calibrate_1125542_31.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/cf79c1d8-1456-41d8-a784-34fd6545652a_calibrate_1125542_31.stderr parsl.dataflow.dflow INFO: Task 535 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c25dceda8cd144d6bf4689440e633da with result from task 535 parsl.dataflow.dflow DEBUG: Task 535 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 536 has memoization hash 011998f4c9db109f20ef50e41a9c7830 parsl.dataflow.memoization INFO: Task 536 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811e54540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:43 proc_lsst.multi:146 [INFO] found job 16889859 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889859 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.dataflow.dflow INFO: Parsl task 536 try 0 launched on executor multi with executor id 549 parsl.dataflow.dflow INFO: Standard output for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/058b21c6-9ede-4404-b39a-ed241d6f9d17_calibrate_1125543_25.stdout parsl.dataflow.dflow INFO: Standard error for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/058b21c6-9ede-4404-b39a-ed241d6f9d17_calibrate_1125543_25.stderr parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/0d5f1499-5ad9-4ece-86c8-c546cf8259ef_characterizeImage_1125543_25.stdout parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/0d5f1499-5ad9-4ece-86c8-c546cf8259ef_characterizeImage_1125543_25.stderr parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c6b9bcf6f5275e7025673f6c760f0dc with result from task 8 parsl.dataflow.dflow DEBUG: Task 8 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/739003f2-42dd-4236-a694-27598d980cc4_calibrate_1125542_38.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/739003f2-42dd-4236-a694-27598d980cc4_calibrate_1125542_38.stderr parsl.dataflow.dflow INFO: Task 466 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5e8e5c5b3a2c25685fb0a9ec9646107 with result from task 466 parsl.dataflow.dflow DEBUG: Task 466 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 467 has memoization hash aaf51429e88ad385ae52f491b8bfc155 parsl.dataflow.memoization INFO: Task 467 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbf7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 467 try 0 launched on executor multi with executor id 550 parsl.dataflow.dflow INFO: Standard output for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/799e1cd1-c494-4894-b241-41dbbec6f25d_calibrate_1125543_49.stdout parsl.dataflow.dflow INFO: Standard error for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/799e1cd1-c494-4894-b241-41dbbec6f25d_calibrate_1125543_49.stderr parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/0eaca5d5-2ceb-4095-b1aa-649dc0117180_characterizeImage_1125543_49.stdout parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/0eaca5d5-2ceb-4095-b1aa-649dc0117180_characterizeImage_1125543_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:45 proc_lsst.multi:146 [INFO] found job 16889860 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889860 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.dataflow.dflow INFO: Task 505 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1e650292e1c58fa620202df3660fba0 with result from task 505 parsl.dataflow.dflow DEBUG: Task 505 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 506 has memoization hash 648d6b3bf649cec6f979265d6f5c4d20 parsl.dataflow.memoization INFO: Task 506 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfd800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 506 try 0 launched on executor multi with executor id 551 parsl.dataflow.dflow INFO: Standard output for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/beab32a2-2485-4ec5-8435-6faf4518b881_calibrate_1125542_26.stdout parsl.dataflow.dflow INFO: Standard error for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/beab32a2-2485-4ec5-8435-6faf4518b881_calibrate_1125542_26.stderr parsl.dataflow.dflow INFO: Standard output for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/c58b27c7-64ff-409b-8740-6d80faded686_characterizeImage_1125542_26.stdout parsl.dataflow.dflow INFO: Standard error for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125542/c58b27c7-64ff-409b-8740-6d80faded686_characterizeImage_1125542_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:46 proc_lsst.multi:146 [INFO] found job 16889861 in provider escience_a40 proc_lsst.multi INFO: found job 16889861 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:47 proc_lsst.multi:146 [INFO] found job 16889862 in provider escience_a40 proc_lsst.multi INFO: found job 16889862 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 41 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3618212f7652211d22f212017b251224 with result from task 41 parsl.dataflow.dflow DEBUG: Task 41 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/85404648-4db4-4acd-9d37-7bfeea39527c_calibrate_1125541_59.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/85404648-4db4-4acd-9d37-7bfeea39527c_calibrate_1125541_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:48 proc_lsst.multi:146 [INFO] found job 16889863 in provider escience_a40 proc_lsst.multi INFO: found job 16889863 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 56 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d18b5145681869cdb2f5ade38436e821 with result from task 56 parsl.dataflow.dflow DEBUG: Task 56 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/c559a43c-08ee-4e45-bdcd-1f5e483f1fb5_calibrate_1125541_44.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/c559a43c-08ee-4e45-bdcd-1f5e483f1fb5_calibrate_1125541_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:49 proc_lsst.multi:146 [INFO] found job 16889864 in provider escience_a40 proc_lsst.multi INFO: found job 16889864 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf9c988b222399f81109042b506c923e with result from task 89 parsl.dataflow.dflow DEBUG: Task 89 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/188141af-b21d-45b2-87e9-5314b492275e_calibrate_1125541_33.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/188141af-b21d-45b2-87e9-5314b492275e_calibrate_1125541_33.stderr parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6bf5c63c77bfba96568fb2cd00fcd63d with result from task 59 parsl.dataflow.dflow DEBUG: Task 59 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/69949520-f6f6-4e87-8fa4-02c704e8d595_calibrate_1125541_10.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/69949520-f6f6-4e87-8fa4-02c704e8d595_calibrate_1125541_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:50 proc_lsst.multi:146 [INFO] found job 16889865 in provider escience_a40 proc_lsst.multi INFO: found job 16889865 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:51 proc_lsst.multi:146 [INFO] found job 16889866 in provider escience_a40 proc_lsst.multi INFO: found job 16889866 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 92 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a68c847280705bd81b4606c5cb4bb629 with result from task 92 parsl.dataflow.dflow DEBUG: Task 92 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/5296b656-648d-408c-bbdc-ba228dd0d9c3_calibrate_1125541_28.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/5296b656-648d-408c-bbdc-ba228dd0d9c3_calibrate_1125541_28.stderr parsl.dataflow.dflow INFO: Task 170 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1669faa1d661b0df13ae4af2025466e8 with result from task 170 parsl.dataflow.dflow DEBUG: Task 170 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/a76a74c5-2d68-4aed-abcf-2e47f92e1edd_calibrate_1125541_51.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/a76a74c5-2d68-4aed-abcf-2e47f92e1edd_calibrate_1125541_51.stderr parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aed6556380bf4dec5757f1d521b08431 with result from task 86 parsl.dataflow.dflow DEBUG: Task 86 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/9fc65122-79ab-41e2-9c53-1764893f9833_calibrate_1125543_32.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/9fc65122-79ab-41e2-9c53-1764893f9833_calibrate_1125543_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:52 proc_lsst.multi:146 [INFO] found job 16889867 in provider escience_a40 proc_lsst.multi INFO: found job 16889867 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7dc571e2b8b4dc1876da8e9c7233b8d with result from task 113 parsl.dataflow.dflow DEBUG: Task 113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/410d034b-9d84-45d6-9ea0-d64343b7e5ce_calibrate_1125543_20.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/410d034b-9d84-45d6-9ea0-d64343b7e5ce_calibrate_1125543_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:53 proc_lsst.multi:146 [INFO] found job 16889868 in provider escience_a40 proc_lsst.multi INFO: found job 16889868 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 553 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c77bf4ecacf7555783b02f2ee417cd17 with result from task 553 parsl.dataflow.dflow DEBUG: Task 553 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 554 has memoization hash fa73e044fa0112b357ee95232d6c51b3 parsl.dataflow.memoization INFO: Task 554 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811e55620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 554 try 0 launched on executor multi with executor id 552 parsl.dataflow.dflow INFO: Standard output for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/e60c4ba3-94c0-4a65-baf5-2bc0bd0e0d70_calibrate_1125541_7.stdout parsl.dataflow.dflow INFO: Standard error for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/e60c4ba3-94c0-4a65-baf5-2bc0bd0e0d70_calibrate_1125541_7.stderr parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/e698a0fa-658c-4a4c-8ef1-d67979152518_characterizeImage_1125541_7.stdout parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/e698a0fa-658c-4a4c-8ef1-d67979152518_characterizeImage_1125541_7.stderr parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ce16498d71a0713f69072c2aa289f81 with result from task 44 parsl.dataflow.dflow DEBUG: Task 44 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/e7ed72a0-5d2f-4843-ae78-8bceaaea2489_calibrate_1125543_8.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/e7ed72a0-5d2f-4843-ae78-8bceaaea2489_calibrate_1125543_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:54 proc_lsst.multi:146 [INFO] found job 16889869 in provider escience_a40 proc_lsst.multi INFO: found job 16889869 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 196 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 171f9864b96e34e0839c38f493bf4134 with result from task 196 parsl.dataflow.dflow DEBUG: Task 196 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 197 has memoization hash a90981252c54cd81e54ff51f60d33aad parsl.dataflow.memoization INFO: Task 197 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed5ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 197 try 0 launched on executor multi with executor id 553 parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/95e26b10-bc11-42eb-8d8b-1b1df2e4c7d3_calibrate_1125541_61.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/95e26b10-bc11-42eb-8d8b-1b1df2e4c7d3_calibrate_1125541_61.stderr parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/6bbb65e9-5d7f-4c11-993c-c917c9c63af3_characterizeImage_1125541_61.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/6bbb65e9-5d7f-4c11-993c-c917c9c63af3_characterizeImage_1125541_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:57 proc_lsst.multi:146 [INFO] found job 16889870 in provider escience_a40 proc_lsst.multi INFO: found job 16889870 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 62 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0267a09c85bc8f5e597ab8b7c98325f0 with result from task 62 parsl.dataflow.dflow DEBUG: Task 62 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/0402a9f9-2a34-4b02-8bb0-b1437ee8126b_calibrate_1125541_27.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/0402a9f9-2a34-4b02-8bb0-b1437ee8126b_calibrate_1125541_27.stderr parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f528cbad6fd04dfcdaec8f1ed0bf18a with result from task 47 parsl.dataflow.dflow DEBUG: Task 47 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/ba6ae70d-58e1-45f6-ad0b-6512534f3dc6_calibrate_1125542_40.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/ba6ae70d-58e1-45f6-ad0b-6512534f3dc6_calibrate_1125542_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:28:59 proc_lsst.multi:146 [INFO] found job 16889871 in provider escience_a40 proc_lsst.multi INFO: found job 16889871 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 164 parsl.dataflow.strategy DEBUG: Executor multi has 164 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22643663051856 2024-03-07 10:29:00 proc_lsst.multi:146 [INFO] found job 24403 in provider local proc_lsst.multi INFO: found job 24403 in provider local 2024-03-07 10:29:00 proc_lsst.multi:146 [INFO] found job 16889823 in provider astro proc_lsst.multi INFO: found job 16889823 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:01 proc_lsst.multi:146 [INFO] found job 16889824 in provider astro proc_lsst.multi INFO: found job 16889824 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 2 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66917e4fc20a2757c587932540d955b7 with result from task 2 parsl.dataflow.dflow DEBUG: Task 2 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/66fe5e1a-93cb-4636-95f2-7243aaab6841_calibrate_1125543_62.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/66fe5e1a-93cb-4636-95f2-7243aaab6841_calibrate_1125543_62.stderr parsl.dataflow.dflow INFO: Task 140 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dfd6144bc06f5dd00a6f3b40ca2a2150 with result from task 140 parsl.dataflow.dflow DEBUG: Task 140 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/23a1bacc-1b3c-46da-8122-92bf30499238_calibrate_1125543_9.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/23a1bacc-1b3c-46da-8122-92bf30499238_calibrate_1125543_9.stderr parsl.dataflow.dflow INFO: Task 223 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fee2ced1baba9fd8a15e920029d11aa with result from task 223 parsl.dataflow.dflow DEBUG: Task 223 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 224 has memoization hash afb070044e404309e6aaa0db015f0f55 parsl.dataflow.memoization INFO: Task 224 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811ed7b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 224 try 0 launched on executor multi with executor id 554 parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/90f2424d-6892-46be-b1cc-3c29e5bfea57_calibrate_1125543_48.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/90f2424d-6892-46be-b1cc-3c29e5bfea57_calibrate_1125543_48.stderr parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/572582d7-e2f9-453d-be08-1033d912a776_characterizeImage_1125543_48.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/572582d7-e2f9-453d-be08-1033d912a776_characterizeImage_1125543_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:02 proc_lsst.multi:146 [INFO] found job 16889825 in provider astro proc_lsst.multi INFO: found job 16889825 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:03 proc_lsst.multi:146 [INFO] found job 16889826 in provider astro proc_lsst.multi INFO: found job 16889826 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:04 proc_lsst.multi:146 [INFO] found job 16889827 in provider astro proc_lsst.multi INFO: found job 16889827 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 511 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aeaef5c7a62a93eb4172c41b6ec62d7a with result from task 511 parsl.dataflow.dflow DEBUG: Task 511 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 512 has memoization hash c0eab586ff843f4bafff2fbf5c029f28 parsl.dataflow.memoization INFO: Task 512 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dfec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 512 try 0 launched on executor multi with executor id 555 parsl.dataflow.dflow INFO: Standard output for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/0f41c564-073a-4a2c-8bca-f412e58d62e5_calibrate_1125543_44.stdout parsl.dataflow.dflow INFO: Standard error for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/0f41c564-073a-4a2c-8bca-f412e58d62e5_calibrate_1125543_44.stderr parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/d17c9d41-1c03-4f32-82ad-9daa739510f0_characterizeImage_1125543_44.stdout parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/d17c9d41-1c03-4f32-82ad-9daa739510f0_characterizeImage_1125543_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:05 proc_lsst.multi:146 [INFO] found job 16889828 in provider astro proc_lsst.multi INFO: found job 16889828 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15888dd33db9794b59ccee6c0a66a255 with result from task 95 parsl.dataflow.dflow DEBUG: Task 95 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/d43c382d-709b-4c4a-880d-96263b69f342_calibrate_1125541_37.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/d43c382d-709b-4c4a-880d-96263b69f342_calibrate_1125541_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:06 proc_lsst.multi:146 [INFO] found job 16889829 in provider astro proc_lsst.multi INFO: found job 16889829 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:08 proc_lsst.multi:146 [INFO] found job 16889830 in provider astro proc_lsst.multi INFO: found job 16889830 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:09 proc_lsst.multi:146 [INFO] found job 16889831 in provider astro proc_lsst.multi INFO: found job 16889831 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc496866428dbf2fc98b8202f5168e96 with result from task 107 parsl.dataflow.dflow DEBUG: Task 107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/a9da3e46-9a24-47bf-80da-dc81b0920ca5_calibrate_1125541_62.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/a9da3e46-9a24-47bf-80da-dc81b0920ca5_calibrate_1125541_62.stderr parsl.dataflow.dflow INFO: Task 104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2819f0434648c88dc7d180deb8a11aa3 with result from task 104 parsl.dataflow.dflow DEBUG: Task 104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/d828b9c9-5382-415b-8d6f-35bcb8fd505e_calibrate_1125543_27.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/d828b9c9-5382-415b-8d6f-35bcb8fd505e_calibrate_1125543_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:10 proc_lsst.multi:146 [INFO] found job 16889832 in provider astro proc_lsst.multi INFO: found job 16889832 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 128 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 049fe27de73bb79a6268525202de1a77 with result from task 128 parsl.dataflow.dflow DEBUG: Task 128 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/3814389b-b750-41cb-91f8-79c8acef780a_calibrate_1125542_43.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/3814389b-b750-41cb-91f8-79c8acef780a_calibrate_1125542_43.stderr parsl.dataflow.dflow INFO: Task 50 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee0b17a415a5a4af1b7111e6ea841553 with result from task 50 parsl.dataflow.dflow DEBUG: Task 50 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/ea0297c2-4166-4889-93a0-2934af51af04_calibrate_1125543_26.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/ea0297c2-4166-4889-93a0-2934af51af04_calibrate_1125543_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:11 proc_lsst.multi:146 [INFO] found job 16889833 in provider astro proc_lsst.multi INFO: found job 16889833 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:12 proc_lsst.multi:146 [INFO] found job 16889834 in provider astro proc_lsst.multi INFO: found job 16889834 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:13 proc_lsst.multi:146 [INFO] found job 16889835 in provider astro proc_lsst.multi INFO: found job 16889835 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 550 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48019b1923fb729bbff3b01f5180158f with result from task 550 parsl.dataflow.dflow DEBUG: Task 550 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 551 has memoization hash 1042bc9e708042f24d1e2ee834c843d9 parsl.dataflow.memoization INFO: Task 551 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811e55300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 551 try 0 launched on executor multi with executor id 556 parsl.dataflow.dflow INFO: Standard output for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/37c8a8c6-dd61-4526-8b2e-e180ae2e755a_calibrate_1125541_56.stdout parsl.dataflow.dflow INFO: Standard error for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/37c8a8c6-dd61-4526-8b2e-e180ae2e755a_calibrate_1125541_56.stderr parsl.dataflow.dflow INFO: Standard output for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/f1a82298-bf28-4111-a38a-67f05dbf5b73_characterizeImage_1125541_56.stdout parsl.dataflow.dflow INFO: Standard error for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/f1a82298-bf28-4111-a38a-67f05dbf5b73_characterizeImage_1125541_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:14 proc_lsst.multi:146 [INFO] found job 16889836 in provider astro proc_lsst.multi INFO: found job 16889836 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:15 proc_lsst.multi:146 [INFO] found job 16889837 in provider astro proc_lsst.multi INFO: found job 16889837 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 134 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52b57bea44de99cde1d5f883ef199284 with result from task 134 parsl.dataflow.dflow DEBUG: Task 134 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/f843776c-f938-404a-bf54-b9604651c914_calibrate_1125543_13.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/f843776c-f938-404a-bf54-b9604651c914_calibrate_1125543_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:16 proc_lsst.multi:146 [INFO] found job 16889838 in provider astro proc_lsst.multi INFO: found job 16889838 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:17 proc_lsst.multi:146 [INFO] found job 16889839 in provider astro proc_lsst.multi INFO: found job 16889839 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 98 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79d973e005cb196836076dd99a5efd59 with result from task 98 parsl.dataflow.dflow DEBUG: Task 98 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/ad1d64e1-9f10-4736-b772-fd5954ae0eeb_calibrate_1125543_39.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/ad1d64e1-9f10-4736-b772-fd5954ae0eeb_calibrate_1125543_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:18 proc_lsst.multi:146 [INFO] found job 16889840 in provider astro proc_lsst.multi INFO: found job 16889840 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22087684b7b68cfe8ddbf7081ce0241a with result from task 38 parsl.dataflow.dflow DEBUG: Task 38 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/4be18a14-27a3-4fc5-b1b0-d1376f7a60f3_calibrate_1125543_50.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/4be18a14-27a3-4fc5-b1b0-d1376f7a60f3_calibrate_1125543_50.stderr parsl.dataflow.dflow INFO: Task 125 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e222da0c3baf4a59ccaa46649b7b6f18 with result from task 125 parsl.dataflow.dflow DEBUG: Task 125 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/ec2ef3d7-b3f1-4588-b744-7a163162120d_calibrate_1125541_21.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/ec2ef3d7-b3f1-4588-b744-7a163162120d_calibrate_1125541_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:19 proc_lsst.multi:146 [INFO] found job 16889841 in provider astro proc_lsst.multi INFO: found job 16889841 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 65 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0e9ca845c7f92bedf2886bb08c21369 with result from task 65 parsl.dataflow.dflow DEBUG: Task 65 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/be495a56-3583-4257-8834-aa242c38745e_calibrate_1125543_38.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/be495a56-3583-4257-8834-aa242c38745e_calibrate_1125543_38.stderr parsl.dataflow.dflow INFO: Task 5 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f69301463319e7df252348ee39648366 with result from task 5 parsl.dataflow.dflow DEBUG: Task 5 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/04316636-23c7-443b-905d-36f142c34a44_calibrate_1125543_19.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/04316636-23c7-443b-905d-36f142c34a44_calibrate_1125543_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:20 proc_lsst.multi:146 [INFO] found job 16889842 in provider astro proc_lsst.multi INFO: found job 16889842 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 71 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b97419799a1c5f6c7093d8bbe18349fd with result from task 71 parsl.dataflow.dflow DEBUG: Task 71 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/333fb6c0-26ff-45a2-ac32-787a5f81fd74_calibrate_1125543_42.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/333fb6c0-26ff-45a2-ac32-787a5f81fd74_calibrate_1125543_42.stderr parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92a504ec48269c7e78c506044637db59 with result from task 20 parsl.dataflow.dflow DEBUG: Task 20 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/94ed5159-e9d5-449a-84c6-198d8f415f8c_calibrate_1125543_12.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/94ed5159-e9d5-449a-84c6-198d8f415f8c_calibrate_1125543_12.stderr parsl.dataflow.dflow INFO: Task 158 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eaead28d4de9cdc4173de290705e7e53 with result from task 158 parsl.dataflow.dflow DEBUG: Task 158 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/0f8e6a00-21cf-465b-b1f7-9438757b38f9_calibrate_1125542_15.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/0f8e6a00-21cf-465b-b1f7-9438757b38f9_calibrate_1125542_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:21 proc_lsst.multi:146 [INFO] found job 16889843 in provider astro proc_lsst.multi INFO: found job 16889843 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:22 proc_lsst.multi:146 [INFO] found job 16889844 in provider astro proc_lsst.multi INFO: found job 16889844 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 146 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9267b34e30014cf8c85da849d3a4ff51 with result from task 146 parsl.dataflow.dflow DEBUG: Task 146 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/f183129a-5200-4437-bf44-29e5311c6a4c_calibrate_1125541_42.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/f183129a-5200-4437-bf44-29e5311c6a4c_calibrate_1125541_42.stderr parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b6db47ea26787cb09ea840294dcefaf with result from task 32 parsl.dataflow.dflow DEBUG: Task 32 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/a4e62d32-0ab0-4e18-87f2-946360180960_calibrate_1125542_50.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/a4e62d32-0ab0-4e18-87f2-946360180960_calibrate_1125542_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:23 proc_lsst.multi:146 [INFO] found job 16889845 in provider astro proc_lsst.multi INFO: found job 16889845 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 68 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2876b2e173111f99ef685c249dbc1943 with result from task 68 parsl.dataflow.dflow DEBUG: Task 68 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/15e7f0ac-14f8-4ef7-aa22-6223ebcab5db_calibrate_1125541_32.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/15e7f0ac-14f8-4ef7-aa22-6223ebcab5db_calibrate_1125541_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:24 proc_lsst.multi:146 [INFO] found job 16889846 in provider astro proc_lsst.multi INFO: found job 16889846 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 131 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f08debb9e3ed1e4c15ddca1daff8282 with result from task 131 parsl.dataflow.dflow DEBUG: Task 131 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/74e6718b-4137-407d-afff-bdc071699184_calibrate_1125542_4.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/74e6718b-4137-407d-afff-bdc071699184_calibrate_1125542_4.stderr parsl.dataflow.dflow INFO: Task 35 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec78bc8912a528bd955db41f0e845712 with result from task 35 parsl.dataflow.dflow DEBUG: Task 35 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/f0c8e90c-75c7-4d69-a4a0-2b57b4799dcb_calibrate_1125543_15.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/f0c8e90c-75c7-4d69-a4a0-2b57b4799dcb_calibrate_1125543_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:25 proc_lsst.multi:146 [INFO] found job 16889847 in provider astro proc_lsst.multi INFO: found job 16889847 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 164 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54ff5938b02c8e0e2baac52b0ec02074 with result from task 164 parsl.dataflow.dflow DEBUG: Task 164 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/24c91ae1-5718-4b1b-a4a9-ed2526abef2b_calibrate_1125542_35.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/24c91ae1-5718-4b1b-a4a9-ed2526abef2b_calibrate_1125542_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:26 proc_lsst.multi:146 [INFO] found job 16889848 in provider astro proc_lsst.multi INFO: found job 16889848 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:27 proc_lsst.multi:146 [INFO] found job 16889852 in provider astro proc_lsst.multi INFO: found job 16889852 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 206 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15d13a98d5fda6684c53f94371f1a49c with result from task 206 parsl.dataflow.dflow DEBUG: Task 206 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/df9ed02b-10f5-45e1-8e08-aadfba5612f3_calibrate_1125542_51.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/df9ed02b-10f5-45e1-8e08-aadfba5612f3_calibrate_1125542_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:28 proc_lsst.multi:146 [INFO] found job 16889853 in provider astro proc_lsst.multi INFO: found job 16889853 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 212 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65f7137ec394cc774e0e65656e0e6fec with result from task 212 parsl.dataflow.dflow DEBUG: Task 212 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/424fad7b-d73c-4c6b-9809-d4ee00f184bd_calibrate_1125542_29.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/424fad7b-d73c-4c6b-9809-d4ee00f184bd_calibrate_1125542_29.stderr parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f645c5e039dfb5a33b2777c64b22f40f with result from task 116 parsl.dataflow.dflow DEBUG: Task 116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/30c33530-89d6-4736-ab1e-bfbd5b200567_calibrate_1125542_20.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/30c33530-89d6-4736-ab1e-bfbd5b200567_calibrate_1125542_20.stderr parsl.dataflow.dflow INFO: Task 137 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8dc31811704d7682dea4cc8b74f84ef9 with result from task 137 parsl.dataflow.dflow DEBUG: Task 137 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/c1dd1f8d-c12d-4c7e-a9ca-4bc75e2193ac_calibrate_1125543_24.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/c1dd1f8d-c12d-4c7e-a9ca-4bc75e2193ac_calibrate_1125543_24.stderr parsl.dataflow.dflow INFO: Task 173 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ef270fd3c3d19478c8827c0e837b04a with result from task 173 parsl.dataflow.dflow DEBUG: Task 173 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/16121d9e-6592-4287-b0ff-10f383e5df76_calibrate_1125541_35.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/16121d9e-6592-4287-b0ff-10f383e5df76_calibrate_1125541_35.stderr parsl.dataflow.dflow INFO: Task 80 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8bec3eb215d1bc03e9499e019edc770b with result from task 80 parsl.dataflow.dflow DEBUG: Task 80 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/db1ab732-f56a-4f1b-a6f2-42cb1cc89c90_calibrate_1125541_20.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/db1ab732-f56a-4f1b-a6f2-42cb1cc89c90_calibrate_1125541_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:30 proc_lsst.multi:146 [INFO] found job 16889854 in provider astro proc_lsst.multi INFO: found job 16889854 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 299 try 0 failed parsl.dataflow.dflow INFO: Task 299 marked for retry parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/ee5536ae-32cb-40e4-a48e-0af0c7465f4c_calibrate_1125543_31.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/ee5536ae-32cb-40e4-a48e-0af0c7465f4c_calibrate_1125543_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 299 has memoization hash fc0ac030e1da00991321daa3af83e165 parsl.dataflow.memoization INFO: Task 299 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811f309a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 299 try 1 launched on executor multi with executor id 557 parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/ee5536ae-32cb-40e4-a48e-0af0c7465f4c_calibrate_1125543_31.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/ee5536ae-32cb-40e4-a48e-0af0c7465f4c_calibrate_1125543_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:31 proc_lsst.multi:146 [INFO] found job 16889855 in provider astro proc_lsst.multi INFO: found job 16889855 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 74 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a28d98b8e73c123f162849ebea6b2a93 with result from task 74 parsl.dataflow.dflow DEBUG: Task 74 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/28035d15-a736-4672-b373-54304f1d23c8_calibrate_1125541_12.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/28035d15-a736-4672-b373-54304f1d23c8_calibrate_1125541_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:32 proc_lsst.multi:146 [INFO] found job 16889856 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889856 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:33 proc_lsst.multi:146 [INFO] found job 16889857 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889857 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.dataflow.dflow INFO: Task 439 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d307c01b55192a97ca92e2dd8ad5aab with result from task 439 parsl.dataflow.dflow DEBUG: Task 439 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 440 has memoization hash 92f33b21b341151b0e0e972b56c20f7d parsl.dataflow.memoization INFO: Task 440 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811dbc7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:34 proc_lsst.multi:146 [INFO] found job 16889858 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889858 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.dataflow.dflow INFO: Parsl task 440 try 0 launched on executor multi with executor id 558 parsl.dataflow.dflow INFO: Standard output for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/01823506-bc58-4b9f-bd7e-1a08e1a6f330_calibrate_1125543_61.stdout parsl.dataflow.dflow INFO: Standard error for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/01823506-bc58-4b9f-bd7e-1a08e1a6f330_calibrate_1125543_61.stderr parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/26871cf2-7dbb-4c7e-a21e-6c07a785a5ea_characterizeImage_1125543_61.stdout parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125543/26871cf2-7dbb-4c7e-a21e-6c07a785a5ea_characterizeImage_1125543_61.stderr parsl.dataflow.dflow INFO: Task 194 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 961b4f991e3ad4ef14def08af6fa4748 with result from task 194 parsl.dataflow.dflow DEBUG: Task 194 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/318eb8ea-25f0-41f8-812a-0bd380e2ea03_calibrate_1125542_13.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/318eb8ea-25f0-41f8-812a-0bd380e2ea03_calibrate_1125542_13.stderr parsl.dataflow.dflow INFO: Task 176 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 498d1c97f3a7f329c1ca109c5afdf067 with result from task 176 parsl.dataflow.dflow DEBUG: Task 176 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/c83f31ed-074a-414a-9016-3d2eeeb5cd12_calibrate_1125543_6.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/c83f31ed-074a-414a-9016-3d2eeeb5cd12_calibrate_1125543_6.stderr parsl.dataflow.dflow INFO: Task 77 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 219df6d39d828ae8c630907f3e4e77bc with result from task 77 parsl.dataflow.dflow DEBUG: Task 77 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/2d19fb1f-bb11-496b-9059-280f8d67c19a_calibrate_1125543_35.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/2d19fb1f-bb11-496b-9059-280f8d67c19a_calibrate_1125543_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:37 proc_lsst.multi:146 [INFO] found job 16889859 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889859 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:38 proc_lsst.multi:146 [INFO] found job 16889860 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889860 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.dataflow.dflow INFO: Task 320 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 934ae968b13822a48e30f81b5ba64519 with result from task 320 parsl.dataflow.dflow DEBUG: Task 320 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/b13cde07-ef0f-4dab-b30e-b6bebec2b2c6_calibrate_1125541_31.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/b13cde07-ef0f-4dab-b30e-b6bebec2b2c6_calibrate_1125541_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:39 proc_lsst.multi:146 [INFO] found job 16889861 in provider escience_a40 proc_lsst.multi INFO: found job 16889861 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 83 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a16ca02291dc704b45a85d0098078b62 with result from task 83 parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:40 proc_lsst.multi:146 [INFO] found job 16889862 in provider escience_a40 proc_lsst.multi INFO: found job 16889862 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow DEBUG: Task 83 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/29cd6056-33ae-4847-9124-f714137a8d13_calibrate_1125542_34.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/29cd6056-33ae-4847-9124-f714137a8d13_calibrate_1125542_34.stderr parsl.dataflow.dflow INFO: Task 167 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4a17830d3bd6ddb528c9daa74b997ce with result from task 167 parsl.dataflow.dflow DEBUG: Task 167 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/a0ac1c04-a38a-4aa8-83f9-8f62d2664245_calibrate_1125543_11.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/a0ac1c04-a38a-4aa8-83f9-8f62d2664245_calibrate_1125543_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:41 proc_lsst.multi:146 [INFO] found job 16889863 in provider escience_a40 proc_lsst.multi INFO: found job 16889863 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 245 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44bbc5c5c58794ca9a0c2dacdde4e620 with result from task 245 parsl.dataflow.dflow DEBUG: Task 245 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/0399e33c-846a-4eed-b7fd-6f6908afb2a2_calibrate_1125542_6.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/0399e33c-846a-4eed-b7fd-6f6908afb2a2_calibrate_1125542_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:42 proc_lsst.multi:146 [INFO] found job 16889864 in provider escience_a40 proc_lsst.multi INFO: found job 16889864 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 251 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9122292bce84d98409a5c31e66ee368b with result from task 251 parsl.dataflow.dflow DEBUG: Task 251 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/0ca42d8b-bfc6-4ce6-baa6-b7c8c824c622_calibrate_1125542_49.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/0ca42d8b-bfc6-4ce6-baa6-b7c8c824c622_calibrate_1125542_49.stderr parsl.dataflow.dflow INFO: Task 152 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb7feaaa86e00b59040f7f1d8e51012d with result from task 152 parsl.dataflow.dflow DEBUG: Task 152 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/f12f82d5-0c88-45f9-825c-58c34ae413e3_calibrate_1125542_7.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/f12f82d5-0c88-45f9-825c-58c34ae413e3_calibrate_1125542_7.stderr parsl.dataflow.dflow INFO: Task 179 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d288afe46a9d414c230b708bbcc8810f with result from task 179 parsl.dataflow.dflow DEBUG: Task 179 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/ec05b8dc-753e-4fd4-8f2c-88a18cb78671_calibrate_1125542_10.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/ec05b8dc-753e-4fd4-8f2c-88a18cb78671_calibrate_1125542_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:43 proc_lsst.multi:146 [INFO] found job 16889865 in provider escience_a40 proc_lsst.multi INFO: found job 16889865 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 119 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce95901752377b691a8cbd94097c6ae7 with result from task 119 parsl.dataflow.dflow DEBUG: Task 119 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/e90895ce-cfa1-49df-9e51-8dca3b9269af_calibrate_1125542_52.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/e90895ce-cfa1-49df-9e51-8dca3b9269af_calibrate_1125542_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:44 proc_lsst.multi:146 [INFO] found job 16889866 in provider escience_a40 proc_lsst.multi INFO: found job 16889866 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 155 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6c87fa34524dc46cbf9f72dae9b1972 with result from task 155 parsl.dataflow.dflow DEBUG: Task 155 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/f9aaa9db-8f88-4f3f-ae5a-552c020827e8_calibrate_1125543_41.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/f9aaa9db-8f88-4f3f-ae5a-552c020827e8_calibrate_1125543_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:45 proc_lsst.multi:146 [INFO] found job 16889867 in provider escience_a40 proc_lsst.multi INFO: found job 16889867 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:46 proc_lsst.multi:146 [INFO] found job 16889868 in provider escience_a40 proc_lsst.multi INFO: found job 16889868 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 248 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1c682868af8c561ab99258a4fa29dd3 with result from task 248 parsl.dataflow.dflow DEBUG: Task 248 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/b7cb758b-872e-4338-88af-b9b196345b40_calibrate_1125542_28.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/b7cb758b-872e-4338-88af-b9b196345b40_calibrate_1125542_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:47 proc_lsst.multi:146 [INFO] found job 16889869 in provider escience_a40 proc_lsst.multi INFO: found job 16889869 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 263 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 426a47c9979f968cccd87eb8daa3ea75 with result from task 263 parsl.dataflow.dflow DEBUG: Task 263 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/e083836a-e28f-4f05-88fc-42dd3074a783_calibrate_1125541_9.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/e083836a-e28f-4f05-88fc-42dd3074a783_calibrate_1125541_9.stderr parsl.dataflow.dflow INFO: Task 272 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f68f8d5d74b78476d1f274f04dd64cf with result from task 272 parsl.dataflow.dflow DEBUG: Task 272 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/b21a8ab8-22d7-4bf0-b688-2afd75ac55d3_calibrate_1125543_36.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/b21a8ab8-22d7-4bf0-b688-2afd75ac55d3_calibrate_1125543_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:48 proc_lsst.multi:146 [INFO] found job 16889870 in provider escience_a40 proc_lsst.multi INFO: found job 16889870 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 182 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18543d741b3cc0787362fae532b0e7f6 with result from task 182 parsl.dataflow.dflow DEBUG: Task 182 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/394242a4-72fe-4b0c-9568-81ede1fc6638_calibrate_1125542_42.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/394242a4-72fe-4b0c-9568-81ede1fc6638_calibrate_1125542_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:49 proc_lsst.multi:146 [INFO] found job 16889871 in provider escience_a40 proc_lsst.multi INFO: found job 16889871 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 119 parsl.dataflow.strategy DEBUG: Executor multi has 119 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22643663051856 2024-03-07 10:29:50 proc_lsst.multi:146 [INFO] found job 24403 in provider local proc_lsst.multi INFO: found job 24403 in provider local 2024-03-07 10:29:50 proc_lsst.multi:146 [INFO] found job 16889823 in provider astro proc_lsst.multi INFO: found job 16889823 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:51 proc_lsst.multi:146 [INFO] found job 16889824 in provider astro proc_lsst.multi INFO: found job 16889824 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:52 proc_lsst.multi:146 [INFO] found job 16889825 in provider astro proc_lsst.multi INFO: found job 16889825 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 218 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91e45dd03b52698a0f5e1947c49b2a4e with result from task 218 parsl.dataflow.dflow DEBUG: Task 218 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/caabbf03-6115-4c8f-912b-6616a6f8685c_calibrate_1125542_32.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/caabbf03-6115-4c8f-912b-6616a6f8685c_calibrate_1125542_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:52 proc_lsst.multi:146 [INFO] found job 16889826 in provider astro proc_lsst.multi INFO: found job 16889826 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:53 proc_lsst.multi:146 [INFO] found job 16889827 in provider astro proc_lsst.multi INFO: found job 16889827 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 143 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54af87744a00b01797e89cb17b7f399f with result from task 143 parsl.dataflow.dflow DEBUG: Task 143 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/206123ad-9007-4aee-9724-7082099cc700_calibrate_1125543_18.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/206123ad-9007-4aee-9724-7082099cc700_calibrate_1125543_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:54 proc_lsst.multi:146 [INFO] found job 16889828 in provider astro proc_lsst.multi INFO: found job 16889828 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 305 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10ef93e188ae966b722adab247735c10 with result from task 305 parsl.dataflow.dflow DEBUG: Task 305 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/a4b1b3b7-215c-41da-8c2f-1dc377d36ac1_calibrate_1125541_46.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/a4b1b3b7-215c-41da-8c2f-1dc377d36ac1_calibrate_1125541_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:55 proc_lsst.multi:146 [INFO] found job 16889829 in provider astro proc_lsst.multi INFO: found job 16889829 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:56 proc_lsst.multi:146 [INFO] found job 16889830 in provider astro proc_lsst.multi INFO: found job 16889830 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 215 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9fe7d2aef32b9b38ef0c39b78c8b5ac5 with result from task 215 parsl.dataflow.dflow DEBUG: Task 215 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/0a261d90-affc-40cd-834b-403b14146c44_calibrate_1125542_48.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/0a261d90-affc-40cd-834b-403b14146c44_calibrate_1125542_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:57 proc_lsst.multi:146 [INFO] found job 16889831 in provider astro proc_lsst.multi INFO: found job 16889831 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 257 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 237daa15b91a8ab8c715b61e72e308a2 with result from task 257 parsl.dataflow.dflow DEBUG: Task 257 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/bf116c39-d646-4ab4-98dc-7bf5b62b4589_calibrate_1125542_9.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/bf116c39-d646-4ab4-98dc-7bf5b62b4589_calibrate_1125542_9.stderr parsl.dataflow.dflow INFO: Task 200 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd217515cd6ddbfd4301d0ce827d7a97 with result from task 200 parsl.dataflow.dflow DEBUG: Task 200 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/131b9c3c-38ae-4ccd-b411-5a40952171b9_calibrate_1125543_43.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/131b9c3c-38ae-4ccd-b411-5a40952171b9_calibrate_1125543_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:58 proc_lsst.multi:146 [INFO] found job 16889832 in provider astro proc_lsst.multi INFO: found job 16889832 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 191 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c994feed6074735badc7c7025ca363c6 with result from task 191 parsl.dataflow.dflow DEBUG: Task 191 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/1c6e7fa1-aa9f-4296-88cb-a199a0700c7a_calibrate_1125543_57.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/1c6e7fa1-aa9f-4296-88cb-a199a0700c7a_calibrate_1125543_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:29:59 proc_lsst.multi:146 [INFO] found job 16889833 in provider astro proc_lsst.multi INFO: found job 16889833 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:30:00 proc_lsst.multi:146 [INFO] found job 16889834 in provider astro proc_lsst.multi INFO: found job 16889834 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 209 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc1a28468607ea902a5c39984d4ca151 with result from task 209 parsl.dataflow.dflow DEBUG: Task 209 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/da7f1ae9-85cd-4d1a-9370-1c238e977e29_calibrate_1125541_48.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/da7f1ae9-85cd-4d1a-9370-1c238e977e29_calibrate_1125541_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:30:03 proc_lsst.multi:146 [INFO] found job 16889835 in provider astro proc_lsst.multi INFO: found job 16889835 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 344 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ac16e6ad230784c0723abe530c5420a with result from task 344 parsl.dataflow.dflow DEBUG: Task 344 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/156ab4d8-4858-4963-8e9e-2bf2f18549f1_calibrate_1125541_19.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/156ab4d8-4858-4963-8e9e-2bf2f18549f1_calibrate_1125541_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:30:04 proc_lsst.multi:146 [INFO] found job 16889836 in provider astro proc_lsst.multi INFO: found job 16889836 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 356 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77016307c3deba7d890ededbfa72c528 with result from task 356 parsl.dataflow.dflow DEBUG: Task 356 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/854171dc-f0b5-4c6d-8c16-274725e2ca40_calibrate_1125542_11.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/854171dc-f0b5-4c6d-8c16-274725e2ca40_calibrate_1125542_11.stderr parsl.dataflow.dflow INFO: Task 386 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 658d023ae65a2b88a870e71e0a9741ec with result from task 386 parsl.dataflow.dflow DEBUG: Task 386 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/6f4db56a-ecc1-4d17-b7a2-77a57f5113b1_calibrate_1125542_57.stdout parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/6f4db56a-ecc1-4d17-b7a2-77a57f5113b1_calibrate_1125542_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:30:05 proc_lsst.multi:146 [INFO] found job 16889837 in provider astro proc_lsst.multi INFO: found job 16889837 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 326 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7e5a9d29e460fc2733bf16434a495c8 with result from task 326 parsl.dataflow.dflow DEBUG: Task 326 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/9d6fbc77-8df6-4890-8538-356bd01e68cb_calibrate_1125542_36.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/9d6fbc77-8df6-4890-8538-356bd01e68cb_calibrate_1125542_36.stderr parsl.dataflow.dflow INFO: Task 185 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c1f1bc2aa758c0c7157935c7b244a54 with result from task 185 parsl.dataflow.dflow DEBUG: Task 185 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/f9bdf75b-3953-44b5-8313-5f3b7c7832e2_calibrate_1125543_59.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/f9bdf75b-3953-44b5-8313-5f3b7c7832e2_calibrate_1125543_59.stderr parsl.dataflow.dflow INFO: Task 296 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f87da21828913efc2788ae01db6b6311 with result from task 296 parsl.dataflow.dflow DEBUG: Task 296 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/25a9c9f7-fd8f-44d7-be6e-830536fb0e9f_calibrate_1125541_14.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/25a9c9f7-fd8f-44d7-be6e-830536fb0e9f_calibrate_1125541_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:30:06 proc_lsst.multi:146 [INFO] found job 16889838 in provider astro proc_lsst.multi INFO: found job 16889838 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:30:08 proc_lsst.multi:146 [INFO] found job 16889839 in provider astro proc_lsst.multi INFO: found job 16889839 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:30:09 proc_lsst.multi:146 [INFO] found job 16889840 in provider astro proc_lsst.multi INFO: found job 16889840 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:30:09 proc_lsst.multi:146 [INFO] found job 16889841 in provider astro proc_lsst.multi INFO: found job 16889841 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 242 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32e8d46c707f8a5c72dcbe2c1f7ebdd4 with result from task 242 parsl.dataflow.dflow DEBUG: Task 242 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/080be549-6195-4e31-ac7c-19855332556d_calibrate_1125541_30.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/080be549-6195-4e31-ac7c-19855332556d_calibrate_1125541_30.stderr parsl.dataflow.dflow INFO: Task 284 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0965391c46366213450fef1358614414 with result from task 284 parsl.dataflow.dflow DEBUG: Task 284 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/70af7927-8e58-4621-b672-ba357ff91759_calibrate_1125543_7.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/70af7927-8e58-4621-b672-ba357ff91759_calibrate_1125543_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:30:11 proc_lsst.multi:146 [INFO] found job 16889842 in provider astro proc_lsst.multi INFO: found job 16889842 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 323 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 605fc10a72295dd137beba6b9305ab0f with result from task 323 parsl.dataflow.dflow DEBUG: Task 323 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/5cdaec8a-c14d-414b-8fc7-95b1f69fefc6_calibrate_1125541_36.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/5cdaec8a-c14d-414b-8fc7-95b1f69fefc6_calibrate_1125541_36.stderr parsl.dataflow.dflow INFO: Task 293 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1a6f169a21dda174ac721df66bf2b65 with result from task 293 parsl.dataflow.dflow DEBUG: Task 293 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/8a6fb022-1392-44a9-b134-7869e416d237_calibrate_1125541_55.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/8a6fb022-1392-44a9-b134-7869e416d237_calibrate_1125541_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:30:12 proc_lsst.multi:146 [INFO] found job 16889843 in provider astro proc_lsst.multi INFO: found job 16889843 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:30:12 proc_lsst.multi:146 [INFO] found job 16889844 in provider astro proc_lsst.multi INFO: found job 16889844 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 269 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 155be302b59b5704e536c2539e3656d2 with result from task 269 parsl.dataflow.dflow DEBUG: Task 269 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/a18f5bd3-fb64-48e8-8099-22aada8ddc61_calibrate_1125542_25.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/a18f5bd3-fb64-48e8-8099-22aada8ddc61_calibrate_1125542_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:30:14 proc_lsst.multi:146 [INFO] found job 16889845 in provider astro proc_lsst.multi INFO: found job 16889845 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 203 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93c330dd4d5a298f11fdf45146b24a00 with result from task 203 parsl.dataflow.dflow DEBUG: Task 203 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/2022f50a-96b2-4619-bbbe-a65f1d1bba21_calibrate_1125543_56.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/2022f50a-96b2-4619-bbbe-a65f1d1bba21_calibrate_1125543_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:30:15 proc_lsst.multi:146 [INFO] found job 16889846 in provider astro proc_lsst.multi INFO: found job 16889846 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 329 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e97c6a3c4a52348e6887d63996db333 with result from task 329 parsl.dataflow.dflow DEBUG: Task 329 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/b8a846d0-65e8-45a6-98fe-1794b863e012_calibrate_1125541_41.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/b8a846d0-65e8-45a6-98fe-1794b863e012_calibrate_1125541_41.stderr parsl.dataflow.dflow INFO: Task 311 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8fc594c06499d832b000484d47623742 with result from task 311 parsl.dataflow.dflow DEBUG: Task 311 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/7743cf93-9e5e-4387-97b8-d483a8d6d0db_calibrate_1125542_16.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/7743cf93-9e5e-4387-97b8-d483a8d6d0db_calibrate_1125542_16.stderr parsl.dataflow.dflow INFO: Task 332 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc976f8eecbf906a4109861056912737 with result from task 332 parsl.dataflow.dflow DEBUG: Task 332 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/caf2be89-e4af-4301-a01e-e197bafd21bf_calibrate_1125542_62.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/caf2be89-e4af-4301-a01e-e197bafd21bf_calibrate_1125542_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:30:16 proc_lsst.multi:146 [INFO] found job 16889847 in provider astro proc_lsst.multi INFO: found job 16889847 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 281 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c38778fe7f103a71946258cb137856b5 with result from task 281 parsl.dataflow.dflow DEBUG: Task 281 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/b57dd2bd-ffef-4760-887b-cbd7e7b96acc_calibrate_1125543_54.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/b57dd2bd-ffef-4760-887b-cbd7e7b96acc_calibrate_1125543_54.stderr parsl.dataflow.dflow INFO: Task 266 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4acb2b5c6beed7b1a9f710385ea1e31a with result from task 266 parsl.dataflow.dflow DEBUG: Task 266 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/2f8a14e1-b362-439b-a847-c51df9891559_calibrate_1125542_17.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/2f8a14e1-b362-439b-a847-c51df9891559_calibrate_1125542_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:30:17 proc_lsst.multi:146 [INFO] found job 16889848 in provider astro proc_lsst.multi INFO: found job 16889848 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 314 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6766e81a31de330e6e11dae20eef02bf with result from task 314 parsl.dataflow.dflow DEBUG: Task 314 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/93f1fb2e-899e-4f1a-ae66-686bdad314f6_calibrate_1125541_17.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/93f1fb2e-899e-4f1a-ae66-686bdad314f6_calibrate_1125541_17.stderr parsl.dataflow.dflow INFO: Task 359 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d44b9fe7c72e4ebb4960331480b58cae with result from task 359 parsl.dataflow.dflow DEBUG: Task 359 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/4d0a7e29-1c53-4085-9a20-9dad80c22c99_calibrate_1125542_47.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/4d0a7e29-1c53-4085-9a20-9dad80c22c99_calibrate_1125542_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:30:18 proc_lsst.multi:146 [INFO] found job 16889852 in provider astro proc_lsst.multi INFO: found job 16889852 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:30:18 proc_lsst.multi:146 [INFO] found job 16889853 in provider astro proc_lsst.multi INFO: found job 16889853 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:30:19 proc_lsst.multi:146 [INFO] found job 16889854 in provider astro proc_lsst.multi INFO: found job 16889854 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 275 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7daa86b1344cad050e5e3d55b648dd18 with result from task 275 parsl.dataflow.dflow DEBUG: Task 275 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/b1695b08-1308-4b3a-a428-b9d29750a46b_calibrate_1125541_24.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/b1695b08-1308-4b3a-a428-b9d29750a46b_calibrate_1125541_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:30:20 proc_lsst.multi:146 [INFO] found job 16889855 in provider astro proc_lsst.multi INFO: found job 16889855 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 338 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 601da629c1befe976e468c35b4456dc1 with result from task 338 parsl.dataflow.dflow DEBUG: Task 338 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/9502c041-ae42-4cd4-ba26-4e35f8eb5ba2_calibrate_1125543_3.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/9502c041-ae42-4cd4-ba26-4e35f8eb5ba2_calibrate_1125543_3.stderr parsl.dataflow.dflow INFO: Task 188 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef913d798e94881b2a92e15d944aedf9 with result from task 188 parsl.dataflow.dflow DEBUG: Task 188 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/411ac3a9-2943-4589-9710-a2b7480f0557_calibrate_1125541_43.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/411ac3a9-2943-4589-9710-a2b7480f0557_calibrate_1125541_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:30:21 proc_lsst.multi:146 [INFO] found job 16889856 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889856 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.dataflow.dflow INFO: Task 290 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ead3ab61c3103063384fbc82e410360e with result from task 290 parsl.dataflow.dflow DEBUG: Task 290 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/32c9971d-848d-4768-89ed-c5f0650a7531_calibrate_1125542_53.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/32c9971d-848d-4768-89ed-c5f0650a7531_calibrate_1125542_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:30:22 proc_lsst.multi:146 [INFO] found job 16889857 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889857 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:30:23 proc_lsst.multi:146 [INFO] found job 16889858 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889858 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.dataflow.dflow INFO: Task 398 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e925a06bc2ac31c1759313f8a529a1b6 with result from task 398 parsl.dataflow.dflow DEBUG: Task 398 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/42c5b188-b3ee-412e-a0b5-fe41004cd7a5_calibrate_1125543_33.stdout parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/42c5b188-b3ee-412e-a0b5-fe41004cd7a5_calibrate_1125543_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:30:24 proc_lsst.multi:146 [INFO] found job 16889859 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889859 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.dataflow.dflow INFO: Task 362 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e86a1f644a72548905cb7add3e2d45f5 with result from task 362 parsl.dataflow.dflow DEBUG: Task 362 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/b81d11b5-2d4e-44d7-8b70-83902b24b488_calibrate_1125543_10.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/b81d11b5-2d4e-44d7-8b70-83902b24b488_calibrate_1125543_10.stderr parsl.dataflow.dflow INFO: Task 383 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09e813fe081f8d0504a75721ef50ceda with result from task 383 parsl.dataflow.dflow DEBUG: Task 383 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/6d5869b3-2738-4c2b-9624-c82fbabe332b_calibrate_1125542_5.stdout parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/6d5869b3-2738-4c2b-9624-c82fbabe332b_calibrate_1125542_5.stderr parsl.dataflow.dflow INFO: Task 419 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 412ba548b076590134b7ff7b1f84627e with result from task 419 parsl.dataflow.dflow DEBUG: Task 419 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/3a983f77-ce8c-4606-9294-efd0481846cf_calibrate_1125542_3.stdout parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/3a983f77-ce8c-4606-9294-efd0481846cf_calibrate_1125542_3.stderr parsl.dataflow.dflow INFO: Task 347 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ede881e5d64097e6c73907f544d67d6c with result from task 347 parsl.dataflow.dflow DEBUG: Task 347 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/3fe7e129-3054-4290-b9eb-ff7a7df41c2e_calibrate_1125541_3.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/3fe7e129-3054-4290-b9eb-ff7a7df41c2e_calibrate_1125541_3.stderr parsl.dataflow.dflow INFO: Task 365 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3135d0432eec4fcc28c91c8cf837ea7 with result from task 365 parsl.dataflow.dflow DEBUG: Task 365 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/051dce63-0194-4c6f-95ae-934f0ad1925d_calibrate_1125541_57.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/051dce63-0194-4c6f-95ae-934f0ad1925d_calibrate_1125541_57.stderr parsl.dataflow.dflow INFO: Task 302 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e0014a2f0b6e668fc2314d19e851c37 with result from task 302 parsl.dataflow.dflow DEBUG: Task 302 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/1d97e5ad-fc93-42e6-a0ce-1badf1be756c_calibrate_1125543_22.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/1d97e5ad-fc93-42e6-a0ce-1badf1be756c_calibrate_1125543_22.stderr parsl.dataflow.dflow INFO: Task 422 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83c46b5703f88a715b89300dd48d4558 with result from task 422 parsl.dataflow.dflow DEBUG: Task 422 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/00e0ba00-d95d-41e2-873e-8ea0718fed80_calibrate_1125542_8.stdout parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/00e0ba00-d95d-41e2-873e-8ea0718fed80_calibrate_1125542_8.stderr parsl.dataflow.dflow INFO: Task 254 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 116a4eb20e28b59e8938b4acf321d8d0 with result from task 254 parsl.dataflow.dflow DEBUG: Task 254 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/5b79a585-7777-43cd-82d3-29204b4df691_calibrate_1125541_49.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/5b79a585-7777-43cd-82d3-29204b4df691_calibrate_1125541_49.stderr parsl.channels.local.local WARNING: Execution of command 'squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860'' failed due to Command 'squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860'' timed out after 10 seconds parsl.utils ERROR: Callback threw an exception - logging and proceeding anyway Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/utils.py", line 341, in make_callback self.callback(*self.cb_args) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/job_status_poller.py", line 114, in poll self._update_state() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/job_status_poller.py", line 121, in _update_state item.poll(now) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/job_status_poller.py", line 47, in poll self._status = self._executor.status() ^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/executors/status_handling.py", line 109, in status status = self._make_status_dict(block_ids, self._provider.status(job_ids)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/gscratch/dirac/shared/opt/proc_lsst/python/proc_lsst/multi.py", line 190, in status statuses += self.providers[provider].status([job_id]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/providers/cluster_provider.py", line 142, in status self._status() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/providers/slurm/slurm.py", line 157, in _status retcode, stdout, stderr = self.execute_wait(cmd) ^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/providers/cluster_provider.py", line 83, in execute_wait return self.channel.execute_wait(cmd, t) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/channels/local/local.py", line 71, in execute_wait (stdout, stderr) = proc.communicate(timeout=walltime) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/subprocess.py", line 1209, in communicate stdout, stderr = self._communicate(input, endtime, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/subprocess.py", line 2114, in _communicate self._check_timeout(endtime, orig_timeout, stdout, stderr) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/subprocess.py", line 1253, in _check_timeout raise TimeoutExpired( subprocess.TimeoutExpired: Command 'squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860'' timed out after 10 seconds 2024-03-07 10:30:35 proc_lsst.multi:146 [INFO] found job 24403 in provider local proc_lsst.multi INFO: found job 24403 in provider local parsl.dataflow.dflow INFO: Task 401 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d212ae73f3712e896f29e58668b4b55 with result from task 401 parsl.dataflow.dflow DEBUG: Task 401 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/1379a7d9-9161-4898-81c2-3f929dcb1c76_calibrate_1125541_54.stdout parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/1379a7d9-9161-4898-81c2-3f929dcb1c76_calibrate_1125541_54.stderr parsl.dataflow.dflow INFO: Task 110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1282ff24ed036bcc838775838343243e with result from task 110 parsl.dataflow.dflow DEBUG: Task 110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/2566dd52-8d9e-42a9-9f47-fa2f4b4e93fd_calibrate_1125542_19.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/2566dd52-8d9e-42a9-9f47-fa2f4b4e93fd_calibrate_1125542_19.stderr parsl.dataflow.dflow INFO: Task 425 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed21ef88428eba8414099e6403de71c5 with result from task 425 parsl.dataflow.dflow DEBUG: Task 425 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/00610bf6-80fb-43d3-af39-7fca115e8812_calibrate_1125543_46.stdout parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/00610bf6-80fb-43d3-af39-7fca115e8812_calibrate_1125543_46.stderr parsl.dataflow.dflow INFO: Task 317 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e47fa7209613d19fce44800ada4b9003 with result from task 317 parsl.dataflow.dflow DEBUG: Task 317 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/49527edb-b3f9-4ef5-94b8-8ef0c3e56f9b_calibrate_1125543_37.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/49527edb-b3f9-4ef5-94b8-8ef0c3e56f9b_calibrate_1125543_37.stderr parsl.dataflow.dflow INFO: Task 335 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55e42475bf3f10a38a3a704bea533f4c with result from task 335 parsl.dataflow.dflow DEBUG: Task 335 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/3ab515e3-cb5e-48e7-8f96-c65e93df04d2_calibrate_1125542_18.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/3ab515e3-cb5e-48e7-8f96-c65e93df04d2_calibrate_1125542_18.stderr parsl.dataflow.dflow INFO: Task 260 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 421820a9640a8871e710913cb080f90b with result from task 260 parsl.dataflow.dflow DEBUG: Task 260 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/cded50cc-d711-4b6e-82ca-4a0f22fbaba1_calibrate_1125543_58.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/cded50cc-d711-4b6e-82ca-4a0f22fbaba1_calibrate_1125543_58.stderr 2024-03-07 10:30:42 proc_lsst.multi:146 [INFO] found job 16889823 in provider astro proc_lsst.multi INFO: found job 16889823 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 353 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82e3b293bbb074994630bf92d8b1432a with result from task 353 parsl.dataflow.dflow DEBUG: Task 353 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/8ec53b36-7310-48b0-b694-9f7c120b3999_calibrate_1125543_55.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/8ec53b36-7310-48b0-b694-9f7c120b3999_calibrate_1125543_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:30:43 proc_lsst.multi:146 [INFO] found job 16889824 in provider astro proc_lsst.multi INFO: found job 16889824 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 407 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 188ec5f686e7e35d154cc6a09f08e424 with result from task 407 parsl.dataflow.dflow DEBUG: Task 407 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/71cb81c5-2df9-4168-b88b-1e7591328d98_calibrate_1125541_16.stdout parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/71cb81c5-2df9-4168-b88b-1e7591328d98_calibrate_1125541_16.stderr parsl.dataflow.dflow INFO: Task 413 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 301263a3a278f2dfe93f22c3aaeebf97 with result from task 413 parsl.dataflow.dflow DEBUG: Task 413 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/d93752b0-8519-41f8-93de-b89ab204bc6f_calibrate_1125541_39.stdout parsl.dataflow.dflow INFO: Standard error for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/d93752b0-8519-41f8-93de-b89ab204bc6f_calibrate_1125541_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:30:44 proc_lsst.multi:146 [INFO] found job 16889825 in provider astro proc_lsst.multi INFO: found job 16889825 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 443 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5826ee3b8570c633a345466e2a7d1746 with result from task 443 parsl.dataflow.dflow DEBUG: Task 443 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/5061ba44-d555-4590-a063-2fb27ef1276d_calibrate_1125541_45.stdout parsl.dataflow.dflow INFO: Standard error for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/5061ba44-d555-4590-a063-2fb27ef1276d_calibrate_1125541_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:30:45 proc_lsst.multi:146 [INFO] found job 16889826 in provider astro proc_lsst.multi INFO: found job 16889826 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:30:46 proc_lsst.multi:146 [INFO] found job 16889827 in provider astro proc_lsst.multi INFO: found job 16889827 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 374 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 297dc023ca432ff172c7f488114ccd5e with result from task 374 parsl.dataflow.dflow DEBUG: Task 374 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/28dc7b19-ae4a-47c0-a638-dd7e31bfa6d8_calibrate_1125542_39.stdout parsl.dataflow.dflow INFO: Standard error for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/28dc7b19-ae4a-47c0-a638-dd7e31bfa6d8_calibrate_1125542_39.stderr parsl.dataflow.dflow INFO: Task 371 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d557ce68e0dce6d2c2c2a2d8817bb2f6 with result from task 371 parsl.dataflow.dflow DEBUG: Task 371 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/d22bb2e6-5f1d-4e52-8cb6-46d2fcda3bf4_calibrate_1125541_53.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/d22bb2e6-5f1d-4e52-8cb6-46d2fcda3bf4_calibrate_1125541_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:30:47 proc_lsst.multi:146 [INFO] found job 16889828 in provider astro proc_lsst.multi INFO: found job 16889828 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:30:48 proc_lsst.multi:146 [INFO] found job 16889829 in provider astro proc_lsst.multi INFO: found job 16889829 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 404 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d7f92741e28e556bc096b319d7c62f6 with result from task 404 parsl.dataflow.dflow DEBUG: Task 404 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/1a025eeb-9a55-49e9-bdd9-3ac3fed7f95f_calibrate_1125542_59.stdout parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/1a025eeb-9a55-49e9-bdd9-3ac3fed7f95f_calibrate_1125542_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:30:49 proc_lsst.multi:146 [INFO] found job 16889830 in provider astro proc_lsst.multi INFO: found job 16889830 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 287 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1efd37cb7e953ae3e421a65fddfd5cf6 with result from task 287 parsl.dataflow.dflow DEBUG: Task 287 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/833b3417-9ae9-482e-8edb-211f94d8cb5d_calibrate_1125543_17.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/833b3417-9ae9-482e-8edb-211f94d8cb5d_calibrate_1125543_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:30:50 proc_lsst.multi:146 [INFO] found job 16889831 in provider astro proc_lsst.multi INFO: found job 16889831 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 350 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd1b211c25987c30978d0b3018bf520d with result from task 350 parsl.dataflow.dflow DEBUG: Task 350 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/6a9e301d-6b5f-476b-aed4-ae9f8f583a19_calibrate_1125542_37.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/6a9e301d-6b5f-476b-aed4-ae9f8f583a19_calibrate_1125542_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:30:51 proc_lsst.multi:146 [INFO] found job 16889832 in provider astro proc_lsst.multi INFO: found job 16889832 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:30:52 proc_lsst.multi:146 [INFO] found job 16889833 in provider astro proc_lsst.multi INFO: found job 16889833 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:30:53 proc_lsst.multi:146 [INFO] found job 16889834 in provider astro proc_lsst.multi INFO: found job 16889834 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 464 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2de0f5afc2beeb1e9c006046b1987782 with result from task 464 parsl.dataflow.dflow DEBUG: Task 464 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/5d5a93d5-3bbe-46f0-bff9-e57a6ee77e80_calibrate_1125543_21.stdout parsl.dataflow.dflow INFO: Standard error for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/5d5a93d5-3bbe-46f0-bff9-e57a6ee77e80_calibrate_1125543_21.stderr parsl.dataflow.dflow INFO: Task 341 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c6781a2f67966abb278db45cc98e608 with result from task 341 parsl.dataflow.dflow DEBUG: Task 341 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/9c536bc0-a8ab-4d33-93d8-9d4a872ae3ca_calibrate_1125543_28.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/9c536bc0-a8ab-4d33-93d8-9d4a872ae3ca_calibrate_1125543_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:30:54 proc_lsst.multi:146 [INFO] found job 16889835 in provider astro proc_lsst.multi INFO: found job 16889835 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 434 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e84699c962f373190a07aff13ee7d26c with result from task 434 parsl.dataflow.dflow DEBUG: Task 434 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/df7d917f-84e4-44b6-beaa-e2c05b19e48a_calibrate_1125542_45.stdout parsl.dataflow.dflow INFO: Standard error for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/df7d917f-84e4-44b6-beaa-e2c05b19e48a_calibrate_1125542_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:30:55 proc_lsst.multi:146 [INFO] found job 16889836 in provider astro proc_lsst.multi INFO: found job 16889836 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 368 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b2afa33f2fcc2a08d7d83f140e14d84 with result from task 368 parsl.dataflow.dflow DEBUG: Task 368 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/62e59b8c-92be-4122-804a-ccc12066cb6c_calibrate_1125543_5.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/62e59b8c-92be-4122-804a-ccc12066cb6c_calibrate_1125543_5.stderr parsl.dataflow.dflow INFO: Task 227 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51ba899a2b91877170a9cc6835ac13f2 with result from task 227 parsl.dataflow.dflow DEBUG: Task 227 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/2839e1e5-f342-419d-97a0-4ff5a8f632e3_calibrate_1125543_51.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/2839e1e5-f342-419d-97a0-4ff5a8f632e3_calibrate_1125543_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:30:56 proc_lsst.multi:146 [INFO] found job 16889837 in provider astro proc_lsst.multi INFO: found job 16889837 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:30:57 proc_lsst.multi:146 [INFO] found job 16889838 in provider astro proc_lsst.multi INFO: found job 16889838 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:30:58 proc_lsst.multi:146 [INFO] found job 16889839 in provider astro proc_lsst.multi INFO: found job 16889839 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 392 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4ba2b53aa4d32787075461c5c5c3576 with result from task 392 parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:30:59 proc_lsst.multi:146 [INFO] found job 16889840 in provider astro proc_lsst.multi INFO: found job 16889840 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow DEBUG: Task 392 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/04b39646-b7f5-4aab-8c05-adc9fb8f63b0_calibrate_1125543_23.stdout parsl.dataflow.dflow INFO: Standard error for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/04b39646-b7f5-4aab-8c05-adc9fb8f63b0_calibrate_1125543_23.stderr parsl.dataflow.dflow INFO: Task 431 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a030bcf8e07ae61f38cd4491cb05635a with result from task 431 parsl.dataflow.dflow DEBUG: Task 431 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/8ffd09ff-c100-48bd-8e03-75beed597f99_calibrate_1125541_23.stdout parsl.dataflow.dflow INFO: Standard error for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/8ffd09ff-c100-48bd-8e03-75beed597f99_calibrate_1125541_23.stderr parsl.dataflow.dflow INFO: Task 473 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6aeefb232b2f2843b91fda495527299a with result from task 473 parsl.dataflow.dflow DEBUG: Task 473 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/2b0afa68-bc68-4100-8dd0-325d982f2224_calibrate_1125542_41.stdout parsl.dataflow.dflow INFO: Standard error for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/2b0afa68-bc68-4100-8dd0-325d982f2224_calibrate_1125542_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:00 proc_lsst.multi:146 [INFO] found job 16889841 in provider astro proc_lsst.multi INFO: found job 16889841 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:01 proc_lsst.multi:146 [INFO] found job 16889842 in provider astro proc_lsst.multi INFO: found job 16889842 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:02 proc_lsst.multi:146 [INFO] found job 16889843 in provider astro proc_lsst.multi INFO: found job 16889843 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 416 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c111e3b22e1cbf7595265ca89beecf2f with result from task 416 parsl.dataflow.dflow DEBUG: Task 416 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/964ecb9d-8ddb-4554-acb1-0a918ae11a19_calibrate_1125542_55.stdout parsl.dataflow.dflow INFO: Standard error for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/964ecb9d-8ddb-4554-acb1-0a918ae11a19_calibrate_1125542_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:03 proc_lsst.multi:146 [INFO] found job 16889844 in provider astro proc_lsst.multi INFO: found job 16889844 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:04 proc_lsst.multi:146 [INFO] found job 16889845 in provider astro proc_lsst.multi INFO: found job 16889845 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 485 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 109e37cce79b23cffb552b7b1dd0e23a with result from task 485 parsl.dataflow.dflow DEBUG: Task 485 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/307330e0-a2b3-4b96-aea4-7778e5a6a9fa_calibrate_1125543_53.stdout parsl.dataflow.dflow INFO: Standard error for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/307330e0-a2b3-4b96-aea4-7778e5a6a9fa_calibrate_1125543_53.stderr parsl.dataflow.dflow INFO: Task 458 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5983840e2833393494bc57aadd309b15 with result from task 458 parsl.dataflow.dflow DEBUG: Task 458 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/a295e6ed-4da2-4ef3-8a22-a19b68c7e206_calibrate_1125541_34.stdout parsl.dataflow.dflow INFO: Standard error for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/a295e6ed-4da2-4ef3-8a22-a19b68c7e206_calibrate_1125541_34.stderr parsl.dataflow.dflow INFO: Task 389 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 74d6b35eb815e13690d225089fd481e4 with result from task 389 parsl.dataflow.dflow DEBUG: Task 389 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/9258693f-fb73-4ad7-9f52-e6f6ccdf7983_calibrate_1125541_47.stdout parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/9258693f-fb73-4ad7-9f52-e6f6ccdf7983_calibrate_1125541_47.stderr parsl.dataflow.dflow INFO: Task 437 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d92e8ec53db1fad8fb019381e9a9f75 with result from task 437 parsl.dataflow.dflow DEBUG: Task 437 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/787c3c0e-0222-410a-8434-ce938699af47_calibrate_1125543_45.stdout parsl.dataflow.dflow INFO: Standard error for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/787c3c0e-0222-410a-8434-ce938699af47_calibrate_1125543_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:04 proc_lsst.multi:146 [INFO] found job 16889846 in provider astro proc_lsst.multi INFO: found job 16889846 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 455 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 330d333c2ad198b43b45b08f72f682cd with result from task 455 parsl.dataflow.dflow DEBUG: Task 455 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/172646fa-d961-4879-90d0-3e5beec93c5f_calibrate_1125541_18.stdout parsl.dataflow.dflow INFO: Standard error for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/172646fa-d961-4879-90d0-3e5beec93c5f_calibrate_1125541_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:05 proc_lsst.multi:146 [INFO] found job 16889847 in provider astro proc_lsst.multi INFO: found job 16889847 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 428 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a4a9abe62a1d966abec37dd7b282fc9 with result from task 428 parsl.dataflow.dflow DEBUG: Task 428 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/133fca33-f817-4e66-ad73-3140723d0224_calibrate_1125541_60.stdout parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/133fca33-f817-4e66-ad73-3140723d0224_calibrate_1125541_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:06 proc_lsst.multi:146 [INFO] found job 16889848 in provider astro proc_lsst.multi INFO: found job 16889848 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 230 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f2a9983ab3290463434f91e88ec3f3d with result from task 230 parsl.dataflow.dflow DEBUG: Task 230 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/1fa7b20e-b7e9-44ee-94f1-c81caa1a7865_calibrate_1125542_12.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/1fa7b20e-b7e9-44ee-94f1-c81caa1a7865_calibrate_1125542_12.stderr parsl.dataflow.dflow INFO: Task 380 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1fbc30d4f400d1d7983ec3d0b7f16d4e with result from task 380 parsl.dataflow.dflow DEBUG: Task 380 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/26488ece-6316-4f52-ac74-b4e52becc16b_calibrate_1125543_30.stdout parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/26488ece-6316-4f52-ac74-b4e52becc16b_calibrate_1125543_30.stderr parsl.dataflow.dflow INFO: Task 395 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b874973087c6dead363d0571e103191 with result from task 395 parsl.dataflow.dflow DEBUG: Task 395 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/3cb6f405-61b6-42db-b91f-fed48166d2cc_calibrate_1125543_29.stdout parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/3cb6f405-61b6-42db-b91f-fed48166d2cc_calibrate_1125543_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:09 proc_lsst.multi:146 [INFO] found job 16889852 in provider astro proc_lsst.multi INFO: found job 16889852 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:11 proc_lsst.multi:146 [INFO] found job 16889853 in provider astro proc_lsst.multi INFO: found job 16889853 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:12 proc_lsst.multi:146 [INFO] found job 16889854 in provider astro proc_lsst.multi INFO: found job 16889854 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:13 proc_lsst.multi:146 [INFO] found job 16889855 in provider astro proc_lsst.multi INFO: found job 16889855 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:14 proc_lsst.multi:146 [INFO] found job 16889856 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889856 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:15 proc_lsst.multi:146 [INFO] found job 16889857 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889857 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:16 proc_lsst.multi:146 [INFO] found job 16889858 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889858 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.dataflow.dflow INFO: Task 527 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7452604f01c6273d891ec59982cb15b1 with result from task 527 parsl.dataflow.dflow DEBUG: Task 527 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/6b53a997-35e5-4955-a904-f122bac7d88e_calibrate_1125541_5.stdout parsl.dataflow.dflow INFO: Standard error for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/6b53a997-35e5-4955-a904-f122bac7d88e_calibrate_1125541_5.stderr parsl.dataflow.dflow INFO: Task 476 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8666d5f99a4009eab6ed7bd02213e69d with result from task 476 parsl.dataflow.dflow DEBUG: Task 476 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/9383e3d5-afff-4c79-90d2-701188178099_calibrate_1125543_60.stdout parsl.dataflow.dflow INFO: Standard error for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/9383e3d5-afff-4c79-90d2-701188178099_calibrate_1125543_60.stderr parsl.dataflow.dflow INFO: Task 308 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1349c74f40eaac4f86b2a8134a055bfe with result from task 308 parsl.dataflow.dflow DEBUG: Task 308 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/592678ee-00b7-41a7-9077-a03f23eb14d5_calibrate_1125542_1.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/592678ee-00b7-41a7-9077-a03f23eb14d5_calibrate_1125542_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:17 proc_lsst.multi:146 [INFO] found job 16889859 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889859 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:18 proc_lsst.multi:146 [INFO] found job 16889860 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889860 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.dataflow.dflow INFO: Task 470 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a34200d2ed1aa4ccf9c0b73a86addd97 with result from task 470 parsl.dataflow.dflow DEBUG: Task 470 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/97649eeb-cfc5-493c-89c7-c206c5f8e1b1_calibrate_1125541_25.stdout parsl.dataflow.dflow INFO: Standard error for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/97649eeb-cfc5-493c-89c7-c206c5f8e1b1_calibrate_1125541_25.stderr parsl.dataflow.dflow INFO: Task 233 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 730055ce73ebd4197295599a86b24042 with result from task 233 parsl.dataflow.dflow DEBUG: Task 233 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/b967b554-56bd-4425-aef8-c03d68f5e294_calibrate_1125543_14.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/b967b554-56bd-4425-aef8-c03d68f5e294_calibrate_1125543_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:20 proc_lsst.multi:146 [INFO] found job 16889861 in provider escience_a40 proc_lsst.multi INFO: found job 16889861 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 461 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f74818594d8b0aa29bef92b68c05a31 with result from task 461 parsl.dataflow.dflow DEBUG: Task 461 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/80f0e546-60fd-40d9-a601-877a5c8fd362_calibrate_1125543_47.stdout parsl.dataflow.dflow INFO: Standard error for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/80f0e546-60fd-40d9-a601-877a5c8fd362_calibrate_1125543_47.stderr parsl.dataflow.dflow INFO: Task 449 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbd1f72b40b2d7cd165ea71737913508 with result from task 449 parsl.dataflow.dflow DEBUG: Task 449 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/d0eecf8a-8386-4d9d-997a-fb622f91d4f5_calibrate_1125542_24.stdout parsl.dataflow.dflow INFO: Standard error for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/d0eecf8a-8386-4d9d-997a-fb622f91d4f5_calibrate_1125542_24.stderr parsl.dataflow.dflow INFO: Task 410 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 88bba1d7e90ac5241b244c6a2f354994 with result from task 410 parsl.dataflow.dflow DEBUG: Task 410 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/121ef870-c453-4464-acbf-9fdf96abc8f5_calibrate_1125543_1.stdout parsl.dataflow.dflow INFO: Standard error for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/121ef870-c453-4464-acbf-9fdf96abc8f5_calibrate_1125543_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:21 proc_lsst.multi:146 [INFO] found job 16889862 in provider escience_a40 proc_lsst.multi INFO: found job 16889862 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:22 proc_lsst.multi:146 [INFO] found job 16889863 in provider escience_a40 proc_lsst.multi INFO: found job 16889863 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:23 proc_lsst.multi:146 [INFO] found job 16889864 in provider escience_a40 proc_lsst.multi INFO: found job 16889864 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 446 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a077b7327ee01550a53aa8acb41e645 with result from task 446 parsl.dataflow.dflow DEBUG: Task 446 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/080c34f8-795b-488c-aac6-dc0b278499ff_calibrate_1125541_40.stdout parsl.dataflow.dflow INFO: Standard error for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/080c34f8-795b-488c-aac6-dc0b278499ff_calibrate_1125541_40.stderr parsl.dataflow.dflow INFO: Task 497 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5745b55cb3d90568f2860fb449fc95e0 with result from task 497 parsl.dataflow.dflow DEBUG: Task 497 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/afb2b54b-1c7a-4f11-b4c5-c1ae4f1ad023_calibrate_1125541_38.stdout parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/afb2b54b-1c7a-4f11-b4c5-c1ae4f1ad023_calibrate_1125541_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:24 proc_lsst.multi:146 [INFO] found job 16889865 in provider escience_a40 proc_lsst.multi INFO: found job 16889865 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:25 proc_lsst.multi:146 [INFO] found job 16889866 in provider escience_a40 proc_lsst.multi INFO: found job 16889866 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 236 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5be368f95e430a4e03161888121f3113 with result from task 236 parsl.dataflow.dflow DEBUG: Task 236 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/b7abbf89-a282-4ec2-af58-625b681c2675_calibrate_1125543_34.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/b7abbf89-a282-4ec2-af58-625b681c2675_calibrate_1125543_34.stderr parsl.dataflow.dflow INFO: Task 503 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25723c069fb7bbaa4624e5f840520317 with result from task 503 parsl.dataflow.dflow DEBUG: Task 503 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/695636d6-9c2e-4635-8e89-c70fcabb136c_calibrate_1125541_8.stdout parsl.dataflow.dflow INFO: Standard error for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/695636d6-9c2e-4635-8e89-c70fcabb136c_calibrate_1125541_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:26 proc_lsst.multi:146 [INFO] found job 16889867 in provider escience_a40 proc_lsst.multi INFO: found job 16889867 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 452 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ef8e289f3c878fc88acc7af0726258a with result from task 452 parsl.dataflow.dflow DEBUG: Task 452 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/c1b949d5-d4dc-4aa8-bfef-2ec05ba219d7_calibrate_1125541_15.stdout parsl.dataflow.dflow INFO: Standard error for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/c1b949d5-d4dc-4aa8-bfef-2ec05ba219d7_calibrate_1125541_15.stderr parsl.dataflow.dflow INFO: Task 482 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50a275fac94897b44869ca953ef1155c with result from task 482 parsl.dataflow.dflow DEBUG: Task 482 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/cc7250ee-4d88-44d2-ae8e-4d3be8d82ff1_calibrate_1125541_58.stdout parsl.dataflow.dflow INFO: Standard error for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/cc7250ee-4d88-44d2-ae8e-4d3be8d82ff1_calibrate_1125541_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:27 proc_lsst.multi:146 [INFO] found job 16889868 in provider escience_a40 proc_lsst.multi INFO: found job 16889868 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:28 proc_lsst.multi:146 [INFO] found job 16889869 in provider escience_a40 proc_lsst.multi INFO: found job 16889869 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 377 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7beba6b4634de9ea4dba14ba4a0f7e22 with result from task 377 parsl.dataflow.dflow DEBUG: Task 377 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/b468c896-ba93-41d9-bbc2-29ccfbb419f6_calibrate_1125541_22.stdout parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/b468c896-ba93-41d9-bbc2-29ccfbb419f6_calibrate_1125541_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:29 proc_lsst.multi:146 [INFO] found job 16889870 in provider escience_a40 proc_lsst.multi INFO: found job 16889870 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 221 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f6570d789a5dc41b198dac57958b44e with result from task 221 parsl.dataflow.dflow DEBUG: Task 221 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/1499a73c-ae0d-49c9-9593-844e30467897_calibrate_1125542_23.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/1499a73c-ae0d-49c9-9593-844e30467897_calibrate_1125542_23.stderr parsl.dataflow.dflow INFO: Task 491 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58d8ddd85bd84742aa42fdc637c06011 with result from task 491 parsl.dataflow.dflow DEBUG: Task 491 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/8cf6fe2e-6185-43ba-81fd-640004e3db4e_calibrate_1125542_14.stdout parsl.dataflow.dflow INFO: Standard error for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/8cf6fe2e-6185-43ba-81fd-640004e3db4e_calibrate_1125542_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:30 proc_lsst.multi:146 [INFO] found job 16889871 in provider escience_a40 proc_lsst.multi INFO: found job 16889871 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.dataflow.dflow INFO: Task 494 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fa82d2c70c30fc3f4c885a52843cf76 with result from task 494 parsl.dataflow.dflow DEBUG: Task 494 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/4b9ec996-b49c-44f8-a030-261594729504_calibrate_1125543_40.stdout parsl.dataflow.dflow INFO: Standard error for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/4b9ec996-b49c-44f8-a030-261594729504_calibrate_1125543_40.stderr parsl.dataflow.dflow INFO: Task 239 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13fcda6c490c071363e48a2d27346353 with result from task 239 parsl.dataflow.dflow DEBUG: Task 239 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/9eed6213-652c-4da2-b451-13e113ef7331_calibrate_1125543_4.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/9eed6213-652c-4da2-b451-13e113ef7331_calibrate_1125543_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 27 parsl.dataflow.strategy DEBUG: Executor multi has 27 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22643663051856 2024-03-07 10:31:31 proc_lsst.multi:146 [INFO] found job 24403 in provider local proc_lsst.multi INFO: found job 24403 in provider local 2024-03-07 10:31:31 proc_lsst.multi:146 [INFO] found job 16889823 in provider astro proc_lsst.multi INFO: found job 16889823 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 479 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b388c3d8557731e70f8347b8c0cd73d with result from task 479 parsl.dataflow.dflow DEBUG: Task 479 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/f7a38e55-4344-43b2-9e11-0a645e996697_calibrate_1125542_30.stdout parsl.dataflow.dflow INFO: Standard error for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/f7a38e55-4344-43b2-9e11-0a645e996697_calibrate_1125542_30.stderr parsl.dataflow.dflow INFO: Task 539 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 219608d91b3757f0e1a87959f3c5447e with result from task 539 parsl.dataflow.dflow DEBUG: Task 539 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/c0151ac4-1623-49f7-b930-964b53c76b22_calibrate_1125542_33.stdout parsl.dataflow.dflow INFO: Standard error for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/c0151ac4-1623-49f7-b930-964b53c76b22_calibrate_1125542_33.stderr parsl.dataflow.dflow INFO: Task 518 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42435c7d72036ae416fa890f16ef34c8 with result from task 518 parsl.dataflow.dflow DEBUG: Task 518 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/65f9b44b-689f-4d7e-a90b-18dac112d2c5_calibrate_1125541_13.stdout parsl.dataflow.dflow INFO: Standard error for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/65f9b44b-689f-4d7e-a90b-18dac112d2c5_calibrate_1125541_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:32 proc_lsst.multi:146 [INFO] found job 16889824 in provider astro proc_lsst.multi INFO: found job 16889824 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 521 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 112bb18bd448f11ca4f24477e7e43678 with result from task 521 parsl.dataflow.dflow DEBUG: Task 521 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/1562f373-1270-4d75-980f-042b522c70b8_calibrate_1125542_21.stdout parsl.dataflow.dflow INFO: Standard error for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/1562f373-1270-4d75-980f-042b522c70b8_calibrate_1125542_21.stderr parsl.dataflow.dflow INFO: Task 509 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3cbbe4974867b8aa101ef7a650a710c5 with result from task 509 parsl.dataflow.dflow DEBUG: Task 509 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/a8c15e45-a640-497c-9dbf-c225b268a818_calibrate_1125542_46.stdout parsl.dataflow.dflow INFO: Standard error for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/a8c15e45-a640-497c-9dbf-c225b268a818_calibrate_1125542_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:33 proc_lsst.multi:146 [INFO] found job 16889825 in provider astro proc_lsst.multi INFO: found job 16889825 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 299 try 1 failed parsl.dataflow.dflow ERROR: Task 299 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry fc0ac030e1da00991321daa3af83e165 with result from task 299 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/ee5536ae-32cb-40e4-a48e-0af0c7465f4c_calibrate_1125543_31.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/ee5536ae-32cb-40e4-a48e-0af0c7465f4c_calibrate_1125543_31.stderr parsl.dataflow.dflow INFO: Task 557 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8d5b17ddc4c0d5963dbf8b654b7e731 with result from task 557 parsl.dataflow.dflow DEBUG: Task 557 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/da9481ab-a802-4695-9060-a0077bea28aa_calibrate_1125543_52.stdout parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/da9481ab-a802-4695-9060-a0077bea28aa_calibrate_1125543_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:34 proc_lsst.multi:146 [INFO] found job 16889826 in provider astro proc_lsst.multi INFO: found job 16889826 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:35 proc_lsst.multi:146 [INFO] found job 16889827 in provider astro proc_lsst.multi INFO: found job 16889827 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 542 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5428772dda4759202606a021c581045c with result from task 542 parsl.dataflow.dflow DEBUG: Task 542 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/f306719e-b0b2-4804-b2cf-2812b0e4210d_calibrate_1125542_56.stdout parsl.dataflow.dflow INFO: Standard error for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/f306719e-b0b2-4804-b2cf-2812b0e4210d_calibrate_1125542_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:36 proc_lsst.multi:146 [INFO] found job 16889828 in provider astro proc_lsst.multi INFO: found job 16889828 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 533 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a616c21e9f03f772c829ac1fa2001b2 with result from task 533 parsl.dataflow.dflow DEBUG: Task 533 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/edb6de06-6730-43cf-82d7-0302cf0bbf03_calibrate_1125543_16.stdout parsl.dataflow.dflow INFO: Standard error for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/edb6de06-6730-43cf-82d7-0302cf0bbf03_calibrate_1125543_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:37 proc_lsst.multi:146 [INFO] found job 16889829 in provider astro proc_lsst.multi INFO: found job 16889829 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:37 proc_lsst.multi:146 [INFO] found job 16889830 in provider astro proc_lsst.multi INFO: found job 16889830 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 488 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb6f3c4c7824d4d537293574b59c9dc1 with result from task 488 parsl.dataflow.dflow DEBUG: Task 488 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/3da2371b-6131-428c-a8a1-b0c5ccbb22bd_calibrate_1125542_44.stdout parsl.dataflow.dflow INFO: Standard error for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/3da2371b-6131-428c-a8a1-b0c5ccbb22bd_calibrate_1125542_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:38 proc_lsst.multi:146 [INFO] found job 16889831 in provider astro proc_lsst.multi INFO: found job 16889831 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 515 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cfbbbd05e3762c6a64691f2ca1795a14 with result from task 515 parsl.dataflow.dflow DEBUG: Task 515 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/9f8916fb-0886-4f28-b808-49f5e34922fc_calibrate_1125542_54.stdout parsl.dataflow.dflow INFO: Standard error for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/9f8916fb-0886-4f28-b808-49f5e34922fc_calibrate_1125542_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:38 proc_lsst.multi:146 [INFO] found job 16889832 in provider astro proc_lsst.multi INFO: found job 16889832 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:39 proc_lsst.multi:146 [INFO] found job 16889833 in provider astro proc_lsst.multi INFO: found job 16889833 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:39 proc_lsst.multi:146 [INFO] found job 16889834 in provider astro proc_lsst.multi INFO: found job 16889834 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 500 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8044736169904b069581cea433eace31 with result from task 500 parsl.dataflow.dflow DEBUG: Task 500 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/4a4a81aa-6011-458c-8026-bce73cac2d55_calibrate_1125542_27.stdout parsl.dataflow.dflow INFO: Standard error for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/4a4a81aa-6011-458c-8026-bce73cac2d55_calibrate_1125542_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:39 proc_lsst.multi:146 [INFO] found job 16889835 in provider astro proc_lsst.multi INFO: found job 16889835 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:40 proc_lsst.multi:146 [INFO] found job 16889836 in provider astro proc_lsst.multi INFO: found job 16889836 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:40 proc_lsst.multi:146 [INFO] found job 16889837 in provider astro proc_lsst.multi INFO: found job 16889837 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:40 proc_lsst.multi:146 [INFO] found job 16889838 in provider astro proc_lsst.multi INFO: found job 16889838 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:40 proc_lsst.multi:146 [INFO] found job 16889839 in provider astro proc_lsst.multi INFO: found job 16889839 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:41 proc_lsst.multi:146 [INFO] found job 16889840 in provider astro proc_lsst.multi INFO: found job 16889840 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 161 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1f60174ab10a33a617d5af76b9eddc9 with result from task 161 parsl.dataflow.dflow DEBUG: Task 161 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/ba24f6e0-f977-403f-9928-5b60c650e2a4_calibrate_1125542_61.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/ba24f6e0-f977-403f-9928-5b60c650e2a4_calibrate_1125542_61.stderr parsl.dataflow.dflow INFO: Task 554 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa73e044fa0112b357ee95232d6c51b3 with result from task 554 parsl.dataflow.dflow DEBUG: Task 554 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/e60c4ba3-94c0-4a65-baf5-2bc0bd0e0d70_calibrate_1125541_7.stdout parsl.dataflow.dflow INFO: Standard error for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/e60c4ba3-94c0-4a65-baf5-2bc0bd0e0d70_calibrate_1125541_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:43 proc_lsst.multi:146 [INFO] found job 16889841 in provider astro proc_lsst.multi INFO: found job 16889841 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:43 proc_lsst.multi:146 [INFO] found job 16889842 in provider astro proc_lsst.multi INFO: found job 16889842 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:45 proc_lsst.multi:146 [INFO] found job 16889843 in provider astro proc_lsst.multi INFO: found job 16889843 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:45 proc_lsst.multi:146 [INFO] found job 16889844 in provider astro proc_lsst.multi INFO: found job 16889844 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:45 proc_lsst.multi:146 [INFO] found job 16889845 in provider astro proc_lsst.multi INFO: found job 16889845 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:45 proc_lsst.multi:146 [INFO] found job 16889846 in provider astro proc_lsst.multi INFO: found job 16889846 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:46 proc_lsst.multi:146 [INFO] found job 16889847 in provider astro proc_lsst.multi INFO: found job 16889847 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:46 proc_lsst.multi:146 [INFO] found job 16889848 in provider astro proc_lsst.multi INFO: found job 16889848 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:46 proc_lsst.multi:146 [INFO] found job 16889852 in provider astro proc_lsst.multi INFO: found job 16889852 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:46 proc_lsst.multi:146 [INFO] found job 16889853 in provider astro proc_lsst.multi INFO: found job 16889853 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 551 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1042bc9e708042f24d1e2ee834c843d9 with result from task 551 parsl.dataflow.dflow DEBUG: Task 551 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/37c8a8c6-dd61-4526-8b2e-e180ae2e755a_calibrate_1125541_56.stdout parsl.dataflow.dflow INFO: Standard error for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/37c8a8c6-dd61-4526-8b2e-e180ae2e755a_calibrate_1125541_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:47 proc_lsst.multi:146 [INFO] found job 16889854 in provider astro proc_lsst.multi INFO: found job 16889854 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:47 proc_lsst.multi:146 [INFO] found job 16889855 in provider astro proc_lsst.multi INFO: found job 16889855 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:47 proc_lsst.multi:146 [INFO] found job 16889856 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889856 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:47 proc_lsst.multi:146 [INFO] found job 16889857 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889857 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:48 proc_lsst.multi:146 [INFO] found job 16889858 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889858 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.dataflow.dflow INFO: Task 548 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d9d49bdcb67cb5f3fde97d4e27daff7 with result from task 548 parsl.dataflow.dflow DEBUG: Task 548 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/5002688a-bf9a-4c46-850b-7cd287de3b12_calibrate_1125541_11.stdout parsl.dataflow.dflow INFO: Standard error for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/5002688a-bf9a-4c46-850b-7cd287de3b12_calibrate_1125541_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:48 proc_lsst.multi:146 [INFO] found job 16889859 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889859 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.dataflow.dflow INFO: Task 524 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f45835d1b6caf58e966d6c6e130501d with result from task 524 parsl.dataflow.dflow DEBUG: Task 524 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/275e90cd-7d33-4e6d-9913-dccf02ef1811_calibrate_1125541_52.stdout parsl.dataflow.dflow INFO: Standard error for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/275e90cd-7d33-4e6d-9913-dccf02ef1811_calibrate_1125541_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:48 proc_lsst.multi:146 [INFO] found job 16889860 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889860 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:48 proc_lsst.multi:146 [INFO] found job 16889861 in provider escience_a40 proc_lsst.multi INFO: found job 16889861 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:49 proc_lsst.multi:146 [INFO] found job 16889862 in provider escience_a40 proc_lsst.multi INFO: found job 16889862 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:49 proc_lsst.multi:146 [INFO] found job 16889863 in provider escience_a40 proc_lsst.multi INFO: found job 16889863 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:49 proc_lsst.multi:146 [INFO] found job 16889864 in provider escience_a40 proc_lsst.multi INFO: found job 16889864 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:49 proc_lsst.multi:146 [INFO] found job 16889865 in provider escience_a40 proc_lsst.multi INFO: found job 16889865 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:50 proc_lsst.multi:146 [INFO] found job 16889866 in provider escience_a40 proc_lsst.multi INFO: found job 16889866 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:50 proc_lsst.multi:146 [INFO] found job 16889867 in provider escience_a40 proc_lsst.multi INFO: found job 16889867 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:50 proc_lsst.multi:146 [INFO] found job 16889868 in provider escience_a40 proc_lsst.multi INFO: found job 16889868 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:50 proc_lsst.multi:146 [INFO] found job 16889869 in provider escience_a40 proc_lsst.multi INFO: found job 16889869 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:51 proc_lsst.multi:146 [INFO] found job 16889870 in provider escience_a40 proc_lsst.multi INFO: found job 16889870 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:31:51 proc_lsst.multi:146 [INFO] found job 16889871 in provider escience_a40 proc_lsst.multi INFO: found job 16889871 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 10 parsl.dataflow.strategy DEBUG: Executor multi has 10 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22643663051856 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 10 parsl.dataflow.strategy DEBUG: Executor multi has 10 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22643663051856 parsl.dataflow.dflow INFO: Task 467 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aaf51429e88ad385ae52f491b8bfc155 with result from task 467 parsl.dataflow.dflow DEBUG: Task 467 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/799e1cd1-c494-4894-b241-41dbbec6f25d_calibrate_1125543_49.stdout parsl.dataflow.dflow INFO: Standard error for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/799e1cd1-c494-4894-b241-41dbbec6f25d_calibrate_1125543_49.stderr parsl.dataflow.dflow INFO: Task 506 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 648d6b3bf649cec6f979265d6f5c4d20 with result from task 506 parsl.dataflow.dflow DEBUG: Task 506 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/beab32a2-2485-4ec5-8435-6faf4518b881_calibrate_1125542_26.stdout parsl.dataflow.dflow INFO: Standard error for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/beab32a2-2485-4ec5-8435-6faf4518b881_calibrate_1125542_26.stderr parsl.dataflow.dflow INFO: Task 545 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41e4215446517498e70871a0ce28f6b7 with result from task 545 parsl.dataflow.dflow DEBUG: Task 545 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/86213a17-a98c-409d-81b8-98c344f70438_calibrate_1125541_50.stdout parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/86213a17-a98c-409d-81b8-98c344f70438_calibrate_1125541_50.stderr parsl.dataflow.dflow INFO: Task 224 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry afb070044e404309e6aaa0db015f0f55 with result from task 224 parsl.dataflow.dflow DEBUG: Task 224 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/90f2424d-6892-46be-b1cc-3c29e5bfea57_calibrate_1125543_48.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/90f2424d-6892-46be-b1cc-3c29e5bfea57_calibrate_1125543_48.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 6 parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22643663051856 parsl.dataflow.dflow INFO: Task 536 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 011998f4c9db109f20ef50e41a9c7830 with result from task 536 parsl.dataflow.dflow DEBUG: Task 536 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/058b21c6-9ede-4404-b39a-ed241d6f9d17_calibrate_1125543_25.stdout parsl.dataflow.dflow INFO: Standard error for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/058b21c6-9ede-4404-b39a-ed241d6f9d17_calibrate_1125543_25.stderr parsl.dataflow.dflow INFO: Task 530 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b193802470311522e91f39fae7f0ed91 with result from task 530 parsl.dataflow.dflow DEBUG: Task 530 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/1ab26cb9-bca3-4569-930e-83a8295ac107_calibrate_1125542_22.stdout parsl.dataflow.dflow INFO: Standard error for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125542/1ab26cb9-bca3-4569-930e-83a8295ac107_calibrate_1125542_22.stderr parsl.dataflow.dflow INFO: Task 512 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0eab586ff843f4bafff2fbf5c029f28 with result from task 512 parsl.dataflow.dflow DEBUG: Task 512 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/0f41c564-073a-4a2c-8bca-f412e58d62e5_calibrate_1125543_44.stdout parsl.dataflow.dflow INFO: Standard error for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/0f41c564-073a-4a2c-8bca-f412e58d62e5_calibrate_1125543_44.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22643663051856 2024-03-07 10:32:06 proc_lsst.multi:146 [INFO] found job 24403 in provider local proc_lsst.multi INFO: found job 24403 in provider local 2024-03-07 10:32:06 proc_lsst.multi:146 [INFO] found job 16889823 in provider astro proc_lsst.multi INFO: found job 16889823 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:06 proc_lsst.multi:146 [INFO] found job 16889824 in provider astro proc_lsst.multi INFO: found job 16889824 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:07 proc_lsst.multi:146 [INFO] found job 16889825 in provider astro proc_lsst.multi INFO: found job 16889825 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:07 proc_lsst.multi:146 [INFO] found job 16889826 in provider astro proc_lsst.multi INFO: found job 16889826 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:07 proc_lsst.multi:146 [INFO] found job 16889827 in provider astro proc_lsst.multi INFO: found job 16889827 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:07 proc_lsst.multi:146 [INFO] found job 16889828 in provider astro proc_lsst.multi INFO: found job 16889828 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:07 proc_lsst.multi:146 [INFO] found job 16889829 in provider astro proc_lsst.multi INFO: found job 16889829 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:08 proc_lsst.multi:146 [INFO] found job 16889830 in provider astro proc_lsst.multi INFO: found job 16889830 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:08 proc_lsst.multi:146 [INFO] found job 16889831 in provider astro proc_lsst.multi INFO: found job 16889831 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:08 proc_lsst.multi:146 [INFO] found job 16889832 in provider astro proc_lsst.multi INFO: found job 16889832 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:08 proc_lsst.multi:146 [INFO] found job 16889833 in provider astro proc_lsst.multi INFO: found job 16889833 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:08 proc_lsst.multi:146 [INFO] found job 16889834 in provider astro proc_lsst.multi INFO: found job 16889834 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:09 proc_lsst.multi:146 [INFO] found job 16889835 in provider astro proc_lsst.multi INFO: found job 16889835 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:09 proc_lsst.multi:146 [INFO] found job 16889836 in provider astro proc_lsst.multi INFO: found job 16889836 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:09 proc_lsst.multi:146 [INFO] found job 16889837 in provider astro proc_lsst.multi INFO: found job 16889837 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:09 proc_lsst.multi:146 [INFO] found job 16889838 in provider astro proc_lsst.multi INFO: found job 16889838 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:09 proc_lsst.multi:146 [INFO] found job 16889839 in provider astro proc_lsst.multi INFO: found job 16889839 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:10 proc_lsst.multi:146 [INFO] found job 16889840 in provider astro proc_lsst.multi INFO: found job 16889840 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:10 proc_lsst.multi:146 [INFO] found job 16889841 in provider astro proc_lsst.multi INFO: found job 16889841 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:10 proc_lsst.multi:146 [INFO] found job 16889842 in provider astro proc_lsst.multi INFO: found job 16889842 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:10 proc_lsst.multi:146 [INFO] found job 16889843 in provider astro proc_lsst.multi INFO: found job 16889843 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:10 proc_lsst.multi:146 [INFO] found job 16889844 in provider astro proc_lsst.multi INFO: found job 16889844 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:11 proc_lsst.multi:146 [INFO] found job 16889845 in provider astro proc_lsst.multi INFO: found job 16889845 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:11 proc_lsst.multi:146 [INFO] found job 16889846 in provider astro proc_lsst.multi INFO: found job 16889846 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:11 proc_lsst.multi:146 [INFO] found job 16889847 in provider astro proc_lsst.multi INFO: found job 16889847 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:11 proc_lsst.multi:146 [INFO] found job 16889848 in provider astro proc_lsst.multi INFO: found job 16889848 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:11 proc_lsst.multi:146 [INFO] found job 16889852 in provider astro proc_lsst.multi INFO: found job 16889852 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:11 proc_lsst.multi:146 [INFO] found job 16889853 in provider astro proc_lsst.multi INFO: found job 16889853 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:12 proc_lsst.multi:146 [INFO] found job 16889854 in provider astro proc_lsst.multi INFO: found job 16889854 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:12 proc_lsst.multi:146 [INFO] found job 16889855 in provider astro proc_lsst.multi INFO: found job 16889855 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:12 proc_lsst.multi:146 [INFO] found job 16889856 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889856 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:12 proc_lsst.multi:146 [INFO] found job 16889857 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889857 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:12 proc_lsst.multi:146 [INFO] found job 16889858 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889858 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:13 proc_lsst.multi:146 [INFO] found job 16889859 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889859 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:13 proc_lsst.multi:146 [INFO] found job 16889860 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889860 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:13 proc_lsst.multi:146 [INFO] found job 16889861 in provider escience_a40 proc_lsst.multi INFO: found job 16889861 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:13 proc_lsst.multi:146 [INFO] found job 16889862 in provider escience_a40 proc_lsst.multi INFO: found job 16889862 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:13 proc_lsst.multi:146 [INFO] found job 16889863 in provider escience_a40 proc_lsst.multi INFO: found job 16889863 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:14 proc_lsst.multi:146 [INFO] found job 16889864 in provider escience_a40 proc_lsst.multi INFO: found job 16889864 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:14 proc_lsst.multi:146 [INFO] found job 16889865 in provider escience_a40 proc_lsst.multi INFO: found job 16889865 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:14 proc_lsst.multi:146 [INFO] found job 16889866 in provider escience_a40 proc_lsst.multi INFO: found job 16889866 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:14 proc_lsst.multi:146 [INFO] found job 16889867 in provider escience_a40 proc_lsst.multi INFO: found job 16889867 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:14 proc_lsst.multi:146 [INFO] found job 16889868 in provider escience_a40 proc_lsst.multi INFO: found job 16889868 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:15 proc_lsst.multi:146 [INFO] found job 16889869 in provider escience_a40 proc_lsst.multi INFO: found job 16889869 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:17 proc_lsst.multi:146 [INFO] found job 16889870 in provider escience_a40 proc_lsst.multi INFO: found job 16889870 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:17 proc_lsst.multi:146 [INFO] found job 16889871 in provider escience_a40 proc_lsst.multi INFO: found job 16889871 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22643663051856 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22643663051856 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22643663051856 parsl.dataflow.dflow INFO: Task 440 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92f33b21b341151b0e0e972b56c20f7d with result from task 440 parsl.dataflow.dflow DEBUG: Task 440 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/01823506-bc58-4b9f-bd7e-1a08e1a6f330_calibrate_1125543_61.stdout parsl.dataflow.dflow INFO: Standard error for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125543/01823506-bc58-4b9f-bd7e-1a08e1a6f330_calibrate_1125543_61.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22643663051856 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22643663051856 parsl.dataflow.dflow INFO: Task 197 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a90981252c54cd81e54ff51f60d33aad with result from task 197 parsl.dataflow.dflow DEBUG: Task 197 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/95e26b10-bc11-42eb-8d8b-1b1df2e4c7d3_calibrate_1125541_61.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/95e26b10-bc11-42eb-8d8b-1b1df2e4c7d3_calibrate_1125541_61.stderr 2024-03-07 10:32:37 proc_lsst.multi:146 [INFO] found job 24403 in provider local proc_lsst.multi INFO: found job 24403 in provider local 2024-03-07 10:32:37 proc_lsst.multi:146 [INFO] found job 16889823 in provider astro proc_lsst.multi INFO: found job 16889823 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:37 proc_lsst.multi:146 [INFO] found job 16889824 in provider astro proc_lsst.multi INFO: found job 16889824 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:37 proc_lsst.multi:146 [INFO] found job 16889825 in provider astro proc_lsst.multi INFO: found job 16889825 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:38 proc_lsst.multi:146 [INFO] found job 16889826 in provider astro proc_lsst.multi INFO: found job 16889826 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:38 proc_lsst.multi:146 [INFO] found job 16889827 in provider astro proc_lsst.multi INFO: found job 16889827 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.dataflow.dflow INFO: Task 277 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc4d6bc2babc79270769680177b47097 with result from task 277 parsl.dataflow.dflow DEBUG: Task 277 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 278 has memoization hash 0cd76c83e3f942580b0677527b9dca37 parsl.dataflow.memoization INFO: Task 278 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149811eff380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 278 try 0 launched on executor multi with executor id 559 parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/3afd7ce1-ac8f-4361-888b-a8e4c5e48405_calibrate_1125541_1.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/3afd7ce1-ac8f-4361-888b-a8e4c5e48405_calibrate_1125541_1.stderr parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/dc78f889-881c-4395-9f08-48866636df02_characterizeImage_1125541_1.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/characterizeImage/1125541/dc78f889-881c-4395-9f08-48866636df02_characterizeImage_1125541_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:38 proc_lsst.multi:146 [INFO] found job 16889828 in provider astro proc_lsst.multi INFO: found job 16889828 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:38 proc_lsst.multi:146 [INFO] found job 16889829 in provider astro proc_lsst.multi INFO: found job 16889829 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:38 proc_lsst.multi:146 [INFO] found job 16889830 in provider astro proc_lsst.multi INFO: found job 16889830 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:39 proc_lsst.multi:146 [INFO] found job 16889831 in provider astro proc_lsst.multi INFO: found job 16889831 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:39 proc_lsst.multi:146 [INFO] found job 16889832 in provider astro proc_lsst.multi INFO: found job 16889832 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:39 proc_lsst.multi:146 [INFO] found job 16889833 in provider astro proc_lsst.multi INFO: found job 16889833 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:39 proc_lsst.multi:146 [INFO] found job 16889834 in provider astro proc_lsst.multi INFO: found job 16889834 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:39 proc_lsst.multi:146 [INFO] found job 16889835 in provider astro proc_lsst.multi INFO: found job 16889835 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:39 proc_lsst.multi:146 [INFO] found job 16889836 in provider astro proc_lsst.multi INFO: found job 16889836 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:40 proc_lsst.multi:146 [INFO] found job 16889837 in provider astro proc_lsst.multi INFO: found job 16889837 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:40 proc_lsst.multi:146 [INFO] found job 16889838 in provider astro proc_lsst.multi INFO: found job 16889838 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:40 proc_lsst.multi:146 [INFO] found job 16889839 in provider astro proc_lsst.multi INFO: found job 16889839 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:40 proc_lsst.multi:146 [INFO] found job 16889840 in provider astro proc_lsst.multi INFO: found job 16889840 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:40 proc_lsst.multi:146 [INFO] found job 16889841 in provider astro proc_lsst.multi INFO: found job 16889841 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:40 proc_lsst.multi:146 [INFO] found job 16889842 in provider astro proc_lsst.multi INFO: found job 16889842 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:41 proc_lsst.multi:146 [INFO] found job 16889843 in provider astro proc_lsst.multi INFO: found job 16889843 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:41 proc_lsst.multi:146 [INFO] found job 16889844 in provider astro proc_lsst.multi INFO: found job 16889844 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:41 proc_lsst.multi:146 [INFO] found job 16889845 in provider astro proc_lsst.multi INFO: found job 16889845 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:41 proc_lsst.multi:146 [INFO] found job 16889846 in provider astro proc_lsst.multi INFO: found job 16889846 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:41 proc_lsst.multi:146 [INFO] found job 16889847 in provider astro proc_lsst.multi INFO: found job 16889847 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:42 proc_lsst.multi:146 [INFO] found job 16889848 in provider astro proc_lsst.multi INFO: found job 16889848 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:42 proc_lsst.multi:146 [INFO] found job 16889852 in provider astro proc_lsst.multi INFO: found job 16889852 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:42 proc_lsst.multi:146 [INFO] found job 16889853 in provider astro proc_lsst.multi INFO: found job 16889853 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:42 proc_lsst.multi:146 [INFO] found job 16889854 in provider astro proc_lsst.multi INFO: found job 16889854 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:42 proc_lsst.multi:146 [INFO] found job 16889855 in provider astro proc_lsst.multi INFO: found job 16889855 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:43 proc_lsst.multi:146 [INFO] found job 16889856 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889856 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:43 proc_lsst.multi:146 [INFO] found job 16889857 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889857 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:43 proc_lsst.multi:146 [INFO] found job 16889858 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889858 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:43 proc_lsst.multi:146 [INFO] found job 16889859 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889859 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:43 proc_lsst.multi:146 [INFO] found job 16889860 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889860 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:44 proc_lsst.multi:146 [INFO] found job 16889861 in provider escience_a40 proc_lsst.multi INFO: found job 16889861 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:44 proc_lsst.multi:146 [INFO] found job 16889862 in provider escience_a40 proc_lsst.multi INFO: found job 16889862 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:44 proc_lsst.multi:146 [INFO] found job 16889863 in provider escience_a40 proc_lsst.multi INFO: found job 16889863 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:44 proc_lsst.multi:146 [INFO] found job 16889864 in provider escience_a40 proc_lsst.multi INFO: found job 16889864 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:44 proc_lsst.multi:146 [INFO] found job 16889865 in provider escience_a40 proc_lsst.multi INFO: found job 16889865 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:45 proc_lsst.multi:146 [INFO] found job 16889866 in provider escience_a40 proc_lsst.multi INFO: found job 16889866 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:45 proc_lsst.multi:146 [INFO] found job 16889867 in provider escience_a40 proc_lsst.multi INFO: found job 16889867 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:45 proc_lsst.multi:146 [INFO] found job 16889868 in provider escience_a40 proc_lsst.multi INFO: found job 16889868 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:45 proc_lsst.multi:146 [INFO] found job 16889869 in provider escience_a40 proc_lsst.multi INFO: found job 16889869 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:45 proc_lsst.multi:146 [INFO] found job 16889870 in provider escience_a40 proc_lsst.multi INFO: found job 16889870 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:32:46 proc_lsst.multi:146 [INFO] found job 16889871 in provider escience_a40 proc_lsst.multi INFO: found job 16889871 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22643663051856 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22643663051856 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22643663051856 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22643663051856 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22643663051856 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22643663051856 2024-03-07 10:33:11 proc_lsst.multi:146 [INFO] found job 24403 in provider local proc_lsst.multi INFO: found job 24403 in provider local 2024-03-07 10:33:11 proc_lsst.multi:146 [INFO] found job 16889823 in provider astro proc_lsst.multi INFO: found job 16889823 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:11 proc_lsst.multi:146 [INFO] found job 16889824 in provider astro proc_lsst.multi INFO: found job 16889824 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:11 proc_lsst.multi:146 [INFO] found job 16889825 in provider astro proc_lsst.multi INFO: found job 16889825 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:12 proc_lsst.multi:146 [INFO] found job 16889826 in provider astro proc_lsst.multi INFO: found job 16889826 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:12 proc_lsst.multi:146 [INFO] found job 16889827 in provider astro proc_lsst.multi INFO: found job 16889827 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:12 proc_lsst.multi:146 [INFO] found job 16889828 in provider astro proc_lsst.multi INFO: found job 16889828 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:12 proc_lsst.multi:146 [INFO] found job 16889829 in provider astro proc_lsst.multi INFO: found job 16889829 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:12 proc_lsst.multi:146 [INFO] found job 16889830 in provider astro proc_lsst.multi INFO: found job 16889830 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:13 proc_lsst.multi:146 [INFO] found job 16889831 in provider astro proc_lsst.multi INFO: found job 16889831 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:13 proc_lsst.multi:146 [INFO] found job 16889832 in provider astro proc_lsst.multi INFO: found job 16889832 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:13 proc_lsst.multi:146 [INFO] found job 16889833 in provider astro proc_lsst.multi INFO: found job 16889833 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:13 proc_lsst.multi:146 [INFO] found job 16889834 in provider astro proc_lsst.multi INFO: found job 16889834 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:13 proc_lsst.multi:146 [INFO] found job 16889835 in provider astro proc_lsst.multi INFO: found job 16889835 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:14 proc_lsst.multi:146 [INFO] found job 16889836 in provider astro proc_lsst.multi INFO: found job 16889836 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:14 proc_lsst.multi:146 [INFO] found job 16889837 in provider astro proc_lsst.multi INFO: found job 16889837 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:14 proc_lsst.multi:146 [INFO] found job 16889838 in provider astro proc_lsst.multi INFO: found job 16889838 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:14 proc_lsst.multi:146 [INFO] found job 16889839 in provider astro proc_lsst.multi INFO: found job 16889839 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:14 proc_lsst.multi:146 [INFO] found job 16889840 in provider astro proc_lsst.multi INFO: found job 16889840 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:15 proc_lsst.multi:146 [INFO] found job 16889841 in provider astro proc_lsst.multi INFO: found job 16889841 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:15 proc_lsst.multi:146 [INFO] found job 16889842 in provider astro proc_lsst.multi INFO: found job 16889842 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:15 proc_lsst.multi:146 [INFO] found job 16889843 in provider astro proc_lsst.multi INFO: found job 16889843 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:15 proc_lsst.multi:146 [INFO] found job 16889844 in provider astro proc_lsst.multi INFO: found job 16889844 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:15 proc_lsst.multi:146 [INFO] found job 16889845 in provider astro proc_lsst.multi INFO: found job 16889845 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:16 proc_lsst.multi:146 [INFO] found job 16889846 in provider astro proc_lsst.multi INFO: found job 16889846 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:16 proc_lsst.multi:146 [INFO] found job 16889847 in provider astro proc_lsst.multi INFO: found job 16889847 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:16 proc_lsst.multi:146 [INFO] found job 16889848 in provider astro proc_lsst.multi INFO: found job 16889848 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:16 proc_lsst.multi:146 [INFO] found job 16889852 in provider astro proc_lsst.multi INFO: found job 16889852 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:17 proc_lsst.multi:146 [INFO] found job 16889853 in provider astro proc_lsst.multi INFO: found job 16889853 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:17 proc_lsst.multi:146 [INFO] found job 16889854 in provider astro proc_lsst.multi INFO: found job 16889854 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:17 proc_lsst.multi:146 [INFO] found job 16889855 in provider astro proc_lsst.multi INFO: found job 16889855 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889823,16889824,16889825,16889826,16889827,16889828,16889829,16889830,16889831,16889832,16889833,16889834,16889835,16889836,16889837,16889838,16889839,16889840,16889841,16889842,16889843,16889844,16889845,16889846,16889847,16889848,16889852,16889853,16889854,16889855' parsl.providers.slurm.slurm DEBUG: squeue returned 16889854 R 16889855 R 16889852 R 16889853 R 16889847 R 16889848 R 16889836 R 16889837 R 16889838 R 16889839 R 16889840 R 16889841 R 16889842 R 16889843 R 16889844 R 16889845 R 16889846 R 16889833 R 16889834 R 16889835 R 16889826 R 16889827 R 16889828 R 16889829 R 16889830 R 16889831 R 16889832 R 16889823 R 16889824 R 16889825 R parsl.providers.slurm.slurm DEBUG: Updating job 16889854 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889855 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889852 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889853 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889847 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889848 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889836 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889837 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889838 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889839 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889840 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889841 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889842 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889843 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889844 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889845 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889846 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889833 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889834 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889835 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889828 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889829 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889830 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889831 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889832 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889825 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:17 proc_lsst.multi:146 [INFO] found job 16889856 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889856 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:17 proc_lsst.multi:146 [INFO] found job 16889857 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889857 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:18 proc_lsst.multi:146 [INFO] found job 16889858 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889858 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:18 proc_lsst.multi:146 [INFO] found job 16889859 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889859 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:18 proc_lsst.multi:146 [INFO] found job 16889860 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889860 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889856,16889857,16889858,16889859,16889860' parsl.providers.slurm.slurm DEBUG: squeue returned 16889858 R 16889859 R 16889860 R 16889856 R 16889857 R parsl.providers.slurm.slurm DEBUG: Updating job 16889858 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889859 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889860 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889856 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889857 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:18 proc_lsst.multi:146 [INFO] found job 16889861 in provider escience_a40 proc_lsst.multi INFO: found job 16889861 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:18 proc_lsst.multi:146 [INFO] found job 16889862 in provider escience_a40 proc_lsst.multi INFO: found job 16889862 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:18 proc_lsst.multi:146 [INFO] found job 16889863 in provider escience_a40 proc_lsst.multi INFO: found job 16889863 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:19 proc_lsst.multi:146 [INFO] found job 16889864 in provider escience_a40 proc_lsst.multi INFO: found job 16889864 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:19 proc_lsst.multi:146 [INFO] found job 16889865 in provider escience_a40 proc_lsst.multi INFO: found job 16889865 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:19 proc_lsst.multi:146 [INFO] found job 16889866 in provider escience_a40 proc_lsst.multi INFO: found job 16889866 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:19 proc_lsst.multi:146 [INFO] found job 16889867 in provider escience_a40 proc_lsst.multi INFO: found job 16889867 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:19 proc_lsst.multi:146 [INFO] found job 16889868 in provider escience_a40 proc_lsst.multi INFO: found job 16889868 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:20 proc_lsst.multi:146 [INFO] found job 16889869 in provider escience_a40 proc_lsst.multi INFO: found job 16889869 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:20 proc_lsst.multi:146 [INFO] found job 16889870 in provider escience_a40 proc_lsst.multi INFO: found job 16889870 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING 2024-03-07 10:33:20 proc_lsst.multi:146 [INFO] found job 16889871 in provider escience_a40 proc_lsst.multi INFO: found job 16889871 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16889861,16889862,16889863,16889864,16889865,16889866,16889867,16889868,16889869,16889870,16889871' parsl.providers.slurm.slurm DEBUG: squeue returned 16889868 R 16889869 R 16889870 R 16889871 R 16889865 R 16889866 R 16889867 R 16889862 R 16889863 R 16889864 R 16889861 R parsl.providers.slurm.slurm DEBUG: Updating job 16889868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889862 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889863 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889864 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16889861 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22643663051856 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22643663051856 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22643663051856 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22643663051856 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22643663051856 parsl.dataflow.dflow INFO: Task 278 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0cd76c83e3f942580b0677527b9dca37 with result from task 278 parsl.dataflow.dflow INFO: DFK cleanup initiated parsl.dataflow.dflow INFO: Summary of tasks in DFK: parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0 parsl.dataflow.dflow INFO: Tasks in state States.pending: 0 parsl.dataflow.dflow INFO: Tasks in state States.running: 0 parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 551 parsl.dataflow.dflow DEBUG: Task 278 checkpointed parsl.dataflow.dflow INFO: Tasks in state States.failed: 4 parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 3 parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Tasks in state States.launched: 0 parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/3afd7ce1-ac8f-4361-888b-a8e4c5e48405_calibrate_1125541_1.stdout parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0 parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/submit/DEEP/20220825/B1b/science#step1/20240307T102204Z/logs/calibrate/1125541/3afd7ce1-ac8f-4361-888b-a8e4c5e48405_calibrate_1125541_1.stderr parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0 parsl.dataflow.dflow INFO: Tasks in state States.joining: 0 parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0 parsl.dataflow.dflow INFO: End of summary parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Closing job status poller parsl.dataflow.dflow INFO: Terminated job status poller parsl.dataflow.dflow INFO: Scaling in and shutting down executors parsl.dataflow.dflow INFO: Scaling in executor multi parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=47, block_ids=[] parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 47 blocks parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1ec73fa58fcf parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1ec73fa58fcf parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 087352836edd parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 087352836edd parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b0fdbd1f65c2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b0fdbd1f65c2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 79d1df49b573 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 79d1df49b573 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5c2877e23eb5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5c2877e23eb5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1a75de3e6643 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1a75de3e6643 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 984b27198c8c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 984b27198c8c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 465f07537974 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 465f07537974 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2ab695b44568 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2ab695b44568 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 33cbba840c2c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 33cbba840c2c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 928a897e6045 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 928a897e6045 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8fea8ae053f0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8fea8ae053f0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0ac5ad6db3ca parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0ac5ad6db3ca parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 224a1434f6ab parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 224a1434f6ab parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: af9f0c37d256 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: af9f0c37d256 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d317f93ba3ed parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d317f93ba3ed parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b783fdd34ca2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b783fdd34ca2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0100016e1d80 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0100016e1d80 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 15d1124bef58 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 15d1124bef58 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ceff95362cd0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ceff95362cd0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0494e8a26662 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0494e8a26662 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5efea14f46fc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5efea14f46fc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 552c77ea152e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 552c77ea152e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e1eaf97f362f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e1eaf97f362f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: aa54a8998221 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: aa54a8998221 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9e5a0c90e25d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9e5a0c90e25d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: be939b303883 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: be939b303883 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7f81d9edb588 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7f81d9edb588 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 145048fc340c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 145048fc340c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ff0864c15a3a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ff0864c15a3a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2eef7c120b46 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2eef7c120b46 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d430f7fa54de parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d430f7fa54de parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7168a93134c5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7168a93134c5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d5cba498fca0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d5cba498fca0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: aea820f31b0d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: aea820f31b0d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7b81906a2f4d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7b81906a2f4d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b7bdd5c27588 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b7bdd5c27588 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 97a85f2f31a4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 97a85f2f31a4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f1a4a85b7af7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f1a4a85b7af7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 07139bbe9284 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 07139bbe9284 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 46095842a523 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 46095842a523 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7e410bd038ef parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7e410bd038ef parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6095308ecba1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6095308ecba1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c43120a49e86 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c43120a49e86 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 23d7967ff2ad parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 23d7967ff2ad parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 91789608b33d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 91789608b33d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3af3e311bee5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3af3e311bee5 2024-03-07 10:33:40 proc_lsst.multi:146 [INFO] found job 16889834 in provider astro proc_lsst.multi INFO: found job 16889834 in provider astro 2024-03-07 10:33:40 proc_lsst.multi:201 [INFO] cancelling 16889834 on provider astro proc_lsst.multi INFO: cancelling 16889834 on provider astro 2024-03-07 10:33:40 proc_lsst.multi:146 [INFO] found job 16889836 in provider astro proc_lsst.multi INFO: found job 16889836 in provider astro 2024-03-07 10:33:40 proc_lsst.multi:201 [INFO] cancelling 16889836 on provider astro proc_lsst.multi INFO: cancelling 16889836 on provider astro 2024-03-07 10:33:40 proc_lsst.multi:146 [INFO] found job 16889865 in provider escience_a40 proc_lsst.multi INFO: found job 16889865 in provider escience_a40 2024-03-07 10:33:40 proc_lsst.multi:201 [INFO] cancelling 16889865 on provider escience_a40 proc_lsst.multi INFO: cancelling 16889865 on provider escience_a40 2024-03-07 10:33:40 proc_lsst.multi:146 [INFO] found job 16889860 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889860 in provider escience_rtx6k 2024-03-07 10:33:40 proc_lsst.multi:201 [INFO] cancelling 16889860 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16889860 on provider escience_rtx6k 2024-03-07 10:33:40 proc_lsst.multi:146 [INFO] found job 24403 in provider local proc_lsst.multi INFO: found job 24403 in provider local 2024-03-07 10:33:40 proc_lsst.multi:201 [INFO] cancelling 24403 on provider local proc_lsst.multi INFO: cancelling 24403 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 24403 2024-03-07 10:33:40 proc_lsst.multi:146 [INFO] found job 16889845 in provider astro proc_lsst.multi INFO: found job 16889845 in provider astro 2024-03-07 10:33:40 proc_lsst.multi:201 [INFO] cancelling 16889845 on provider astro proc_lsst.multi INFO: cancelling 16889845 on provider astro 2024-03-07 10:33:40 proc_lsst.multi:146 [INFO] found job 16889847 in provider astro proc_lsst.multi INFO: found job 16889847 in provider astro 2024-03-07 10:33:40 proc_lsst.multi:201 [INFO] cancelling 16889847 on provider astro proc_lsst.multi INFO: cancelling 16889847 on provider astro 2024-03-07 10:33:40 proc_lsst.multi:146 [INFO] found job 16889841 in provider astro proc_lsst.multi INFO: found job 16889841 in provider astro 2024-03-07 10:33:40 proc_lsst.multi:201 [INFO] cancelling 16889841 on provider astro proc_lsst.multi INFO: cancelling 16889841 on provider astro 2024-03-07 10:33:40 proc_lsst.multi:146 [INFO] found job 16889828 in provider astro proc_lsst.multi INFO: found job 16889828 in provider astro 2024-03-07 10:33:40 proc_lsst.multi:201 [INFO] cancelling 16889828 on provider astro proc_lsst.multi INFO: cancelling 16889828 on provider astro 2024-03-07 10:33:40 proc_lsst.multi:146 [INFO] found job 16889838 in provider astro proc_lsst.multi INFO: found job 16889838 in provider astro 2024-03-07 10:33:40 proc_lsst.multi:201 [INFO] cancelling 16889838 on provider astro proc_lsst.multi INFO: cancelling 16889838 on provider astro 2024-03-07 10:33:41 proc_lsst.multi:146 [INFO] found job 16889870 in provider escience_a40 proc_lsst.multi INFO: found job 16889870 in provider escience_a40 2024-03-07 10:33:41 proc_lsst.multi:201 [INFO] cancelling 16889870 on provider escience_a40 proc_lsst.multi INFO: cancelling 16889870 on provider escience_a40 2024-03-07 10:33:41 proc_lsst.multi:146 [INFO] found job 16889831 in provider astro proc_lsst.multi INFO: found job 16889831 in provider astro 2024-03-07 10:33:41 proc_lsst.multi:201 [INFO] cancelling 16889831 on provider astro proc_lsst.multi INFO: cancelling 16889831 on provider astro 2024-03-07 10:33:41 proc_lsst.multi:146 [INFO] found job 16889826 in provider astro proc_lsst.multi INFO: found job 16889826 in provider astro 2024-03-07 10:33:41 proc_lsst.multi:201 [INFO] cancelling 16889826 on provider astro proc_lsst.multi INFO: cancelling 16889826 on provider astro 2024-03-07 10:33:41 proc_lsst.multi:146 [INFO] found job 16889848 in provider astro proc_lsst.multi INFO: found job 16889848 in provider astro 2024-03-07 10:33:41 proc_lsst.multi:201 [INFO] cancelling 16889848 on provider astro proc_lsst.multi INFO: cancelling 16889848 on provider astro 2024-03-07 10:33:41 proc_lsst.multi:146 [INFO] found job 16889855 in provider astro proc_lsst.multi INFO: found job 16889855 in provider astro 2024-03-07 10:33:41 proc_lsst.multi:201 [INFO] cancelling 16889855 on provider astro proc_lsst.multi INFO: cancelling 16889855 on provider astro 2024-03-07 10:33:41 proc_lsst.multi:146 [INFO] found job 16889835 in provider astro proc_lsst.multi INFO: found job 16889835 in provider astro 2024-03-07 10:33:41 proc_lsst.multi:201 [INFO] cancelling 16889835 on provider astro proc_lsst.multi INFO: cancelling 16889835 on provider astro 2024-03-07 10:33:41 proc_lsst.multi:146 [INFO] found job 16889827 in provider astro proc_lsst.multi INFO: found job 16889827 in provider astro 2024-03-07 10:33:41 proc_lsst.multi:201 [INFO] cancelling 16889827 on provider astro proc_lsst.multi INFO: cancelling 16889827 on provider astro 2024-03-07 10:33:41 proc_lsst.multi:146 [INFO] found job 16889832 in provider astro proc_lsst.multi INFO: found job 16889832 in provider astro 2024-03-07 10:33:41 proc_lsst.multi:201 [INFO] cancelling 16889832 on provider astro proc_lsst.multi INFO: cancelling 16889832 on provider astro 2024-03-07 10:33:42 proc_lsst.multi:146 [INFO] found job 16889863 in provider escience_a40 proc_lsst.multi INFO: found job 16889863 in provider escience_a40 2024-03-07 10:33:42 proc_lsst.multi:201 [INFO] cancelling 16889863 on provider escience_a40 proc_lsst.multi INFO: cancelling 16889863 on provider escience_a40 2024-03-07 10:33:42 proc_lsst.multi:146 [INFO] found job 16889859 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889859 in provider escience_rtx6k 2024-03-07 10:33:42 proc_lsst.multi:201 [INFO] cancelling 16889859 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16889859 on provider escience_rtx6k 2024-03-07 10:33:42 proc_lsst.multi:146 [INFO] found job 16889871 in provider escience_a40 proc_lsst.multi INFO: found job 16889871 in provider escience_a40 2024-03-07 10:33:42 proc_lsst.multi:201 [INFO] cancelling 16889871 on provider escience_a40 proc_lsst.multi INFO: cancelling 16889871 on provider escience_a40 2024-03-07 10:33:42 proc_lsst.multi:146 [INFO] found job 16889839 in provider astro proc_lsst.multi INFO: found job 16889839 in provider astro 2024-03-07 10:33:42 proc_lsst.multi:201 [INFO] cancelling 16889839 on provider astro proc_lsst.multi INFO: cancelling 16889839 on provider astro 2024-03-07 10:33:42 proc_lsst.multi:146 [INFO] found job 16889867 in provider escience_a40 proc_lsst.multi INFO: found job 16889867 in provider escience_a40 2024-03-07 10:33:42 proc_lsst.multi:201 [INFO] cancelling 16889867 on provider escience_a40 proc_lsst.multi INFO: cancelling 16889867 on provider escience_a40 2024-03-07 10:33:42 proc_lsst.multi:146 [INFO] found job 16889869 in provider escience_a40 proc_lsst.multi INFO: found job 16889869 in provider escience_a40 2024-03-07 10:33:42 proc_lsst.multi:201 [INFO] cancelling 16889869 on provider escience_a40 proc_lsst.multi INFO: cancelling 16889869 on provider escience_a40 2024-03-07 10:33:42 proc_lsst.multi:146 [INFO] found job 16889853 in provider astro proc_lsst.multi INFO: found job 16889853 in provider astro 2024-03-07 10:33:42 proc_lsst.multi:201 [INFO] cancelling 16889853 on provider astro proc_lsst.multi INFO: cancelling 16889853 on provider astro 2024-03-07 10:33:42 proc_lsst.multi:146 [INFO] found job 16889852 in provider astro proc_lsst.multi INFO: found job 16889852 in provider astro 2024-03-07 10:33:42 proc_lsst.multi:201 [INFO] cancelling 16889852 on provider astro proc_lsst.multi INFO: cancelling 16889852 on provider astro 2024-03-07 10:33:42 proc_lsst.multi:146 [INFO] found job 16889837 in provider astro proc_lsst.multi INFO: found job 16889837 in provider astro 2024-03-07 10:33:42 proc_lsst.multi:201 [INFO] cancelling 16889837 on provider astro proc_lsst.multi INFO: cancelling 16889837 on provider astro 2024-03-07 10:33:43 proc_lsst.multi:146 [INFO] found job 16889842 in provider astro proc_lsst.multi INFO: found job 16889842 in provider astro 2024-03-07 10:33:43 proc_lsst.multi:201 [INFO] cancelling 16889842 on provider astro proc_lsst.multi INFO: cancelling 16889842 on provider astro 2024-03-07 10:33:43 proc_lsst.multi:146 [INFO] found job 16889862 in provider escience_a40 proc_lsst.multi INFO: found job 16889862 in provider escience_a40 2024-03-07 10:33:43 proc_lsst.multi:201 [INFO] cancelling 16889862 on provider escience_a40 proc_lsst.multi INFO: cancelling 16889862 on provider escience_a40 2024-03-07 10:33:43 proc_lsst.multi:146 [INFO] found job 16889861 in provider escience_a40 proc_lsst.multi INFO: found job 16889861 in provider escience_a40 2024-03-07 10:33:43 proc_lsst.multi:201 [INFO] cancelling 16889861 on provider escience_a40 proc_lsst.multi INFO: cancelling 16889861 on provider escience_a40 2024-03-07 10:33:43 proc_lsst.multi:146 [INFO] found job 16889856 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889856 in provider escience_rtx6k 2024-03-07 10:33:43 proc_lsst.multi:201 [INFO] cancelling 16889856 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16889856 on provider escience_rtx6k 2024-03-07 10:33:43 proc_lsst.multi:146 [INFO] found job 16889857 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889857 in provider escience_rtx6k 2024-03-07 10:33:43 proc_lsst.multi:201 [INFO] cancelling 16889857 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16889857 on provider escience_rtx6k 2024-03-07 10:33:43 proc_lsst.multi:146 [INFO] found job 16889840 in provider astro proc_lsst.multi INFO: found job 16889840 in provider astro 2024-03-07 10:33:43 proc_lsst.multi:201 [INFO] cancelling 16889840 on provider astro proc_lsst.multi INFO: cancelling 16889840 on provider astro 2024-03-07 10:33:43 proc_lsst.multi:146 [INFO] found job 16889854 in provider astro proc_lsst.multi INFO: found job 16889854 in provider astro 2024-03-07 10:33:43 proc_lsst.multi:201 [INFO] cancelling 16889854 on provider astro proc_lsst.multi INFO: cancelling 16889854 on provider astro 2024-03-07 10:33:43 proc_lsst.multi:146 [INFO] found job 16889823 in provider astro proc_lsst.multi INFO: found job 16889823 in provider astro 2024-03-07 10:33:43 proc_lsst.multi:201 [INFO] cancelling 16889823 on provider astro proc_lsst.multi INFO: cancelling 16889823 on provider astro 2024-03-07 10:33:43 proc_lsst.multi:146 [INFO] found job 16889825 in provider astro proc_lsst.multi INFO: found job 16889825 in provider astro 2024-03-07 10:33:43 proc_lsst.multi:201 [INFO] cancelling 16889825 on provider astro proc_lsst.multi INFO: cancelling 16889825 on provider astro 2024-03-07 10:33:44 proc_lsst.multi:146 [INFO] found job 16889844 in provider astro proc_lsst.multi INFO: found job 16889844 in provider astro 2024-03-07 10:33:44 proc_lsst.multi:201 [INFO] cancelling 16889844 on provider astro proc_lsst.multi INFO: cancelling 16889844 on provider astro 2024-03-07 10:33:44 proc_lsst.multi:146 [INFO] found job 16889864 in provider escience_a40 proc_lsst.multi INFO: found job 16889864 in provider escience_a40 2024-03-07 10:33:44 proc_lsst.multi:201 [INFO] cancelling 16889864 on provider escience_a40 proc_lsst.multi INFO: cancelling 16889864 on provider escience_a40 2024-03-07 10:33:44 proc_lsst.multi:146 [INFO] found job 16889858 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889858 in provider escience_rtx6k 2024-03-07 10:33:44 proc_lsst.multi:201 [INFO] cancelling 16889858 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16889858 on provider escience_rtx6k 2024-03-07 10:33:44 proc_lsst.multi:146 [INFO] found job 16889868 in provider escience_a40 proc_lsst.multi INFO: found job 16889868 in provider escience_a40 2024-03-07 10:33:44 proc_lsst.multi:201 [INFO] cancelling 16889868 on provider escience_a40 proc_lsst.multi INFO: cancelling 16889868 on provider escience_a40 2024-03-07 10:33:44 proc_lsst.multi:146 [INFO] found job 16889833 in provider astro proc_lsst.multi INFO: found job 16889833 in provider astro 2024-03-07 10:33:44 proc_lsst.multi:201 [INFO] cancelling 16889833 on provider astro proc_lsst.multi INFO: cancelling 16889833 on provider astro 2024-03-07 10:33:44 proc_lsst.multi:146 [INFO] found job 16889830 in provider astro proc_lsst.multi INFO: found job 16889830 in provider astro 2024-03-07 10:33:44 proc_lsst.multi:201 [INFO] cancelling 16889830 on provider astro proc_lsst.multi INFO: cancelling 16889830 on provider astro 2024-03-07 10:33:44 proc_lsst.multi:146 [INFO] found job 16889843 in provider astro proc_lsst.multi INFO: found job 16889843 in provider astro 2024-03-07 10:33:44 proc_lsst.multi:201 [INFO] cancelling 16889843 on provider astro proc_lsst.multi INFO: cancelling 16889843 on provider astro 2024-03-07 10:33:44 proc_lsst.multi:146 [INFO] found job 16889824 in provider astro proc_lsst.multi INFO: found job 16889824 in provider astro 2024-03-07 10:33:44 proc_lsst.multi:201 [INFO] cancelling 16889824 on provider astro proc_lsst.multi INFO: cancelling 16889824 on provider astro 2024-03-07 10:33:45 proc_lsst.multi:146 [INFO] found job 16889866 in provider escience_a40 proc_lsst.multi INFO: found job 16889866 in provider escience_a40 2024-03-07 10:33:45 proc_lsst.multi:201 [INFO] cancelling 16889866 on provider escience_a40 proc_lsst.multi INFO: cancelling 16889866 on provider escience_a40 2024-03-07 10:33:45 proc_lsst.multi:146 [INFO] found job 16889846 in provider astro proc_lsst.multi INFO: found job 16889846 in provider astro 2024-03-07 10:33:45 proc_lsst.multi:201 [INFO] cancelling 16889846 on provider astro proc_lsst.multi INFO: cancelling 16889846 on provider astro 2024-03-07 10:33:45 proc_lsst.multi:146 [INFO] found job 16889829 in provider astro proc_lsst.multi INFO: found job 16889829 in provider astro 2024-03-07 10:33:45 proc_lsst.multi:201 [INFO] cancelling 16889829 on provider astro proc_lsst.multi INFO: cancelling 16889829 on provider astro parsl.dataflow.dflow INFO: Shutting down executor multi 2024-03-07 10:33:45 proc_lsst.multi:40 [INFO] Cancelling all provider resources proc_lsst.multi INFO: Cancelling all provider resources 2024-03-07 10:33:45 proc_lsst.multi:47 [INFO] new jobs since last cancel ['16889823', '16889824', '16889825', '16889826', '16889827', '16889828', '16889829', '16889830', '16889831', '16889832', '16889833', '16889834', '16889835', '16889836', '16889837', '16889838', '16889839', '16889840', '16889841', '16889842', '16889843', '16889844', '16889845', '16889846', '16889847', '16889848', '16889852', '16889853', '16889854', '16889855', '16889861', '16889862', '16889863', '16889864', '16889865', '16889866', '16889867', '16889868', '16889869', '16889870', '16889871', '16889856', '16889857', '16889858', '16889859', '16889860', '24403'] proc_lsst.multi INFO: new jobs since last cancel ['16889823', '16889824', '16889825', '16889826', '16889827', '16889828', '16889829', '16889830', '16889831', '16889832', '16889833', '16889834', '16889835', '16889836', '16889837', '16889838', '16889839', '16889840', '16889841', '16889842', '16889843', '16889844', '16889845', '16889846', '16889847', '16889848', '16889852', '16889853', '16889854', '16889855', '16889861', '16889862', '16889863', '16889864', '16889865', '16889866', '16889867', '16889868', '16889869', '16889870', '16889871', '16889856', '16889857', '16889858', '16889859', '16889860', '24403'] 2024-03-07 10:33:45 proc_lsst.multi:146 [INFO] found job 16889823 in provider astro proc_lsst.multi INFO: found job 16889823 in provider astro 2024-03-07 10:33:45 proc_lsst.multi:201 [INFO] cancelling 16889823 on provider astro proc_lsst.multi INFO: cancelling 16889823 on provider astro 2024-03-07 10:33:45 proc_lsst.multi:146 [INFO] found job 16889824 in provider astro proc_lsst.multi INFO: found job 16889824 in provider astro 2024-03-07 10:33:45 proc_lsst.multi:201 [INFO] cancelling 16889824 on provider astro proc_lsst.multi INFO: cancelling 16889824 on provider astro 2024-03-07 10:33:45 proc_lsst.multi:146 [INFO] found job 16889825 in provider astro proc_lsst.multi INFO: found job 16889825 in provider astro 2024-03-07 10:33:45 proc_lsst.multi:201 [INFO] cancelling 16889825 on provider astro proc_lsst.multi INFO: cancelling 16889825 on provider astro 2024-03-07 10:33:46 proc_lsst.multi:146 [INFO] found job 16889826 in provider astro proc_lsst.multi INFO: found job 16889826 in provider astro 2024-03-07 10:33:46 proc_lsst.multi:201 [INFO] cancelling 16889826 on provider astro proc_lsst.multi INFO: cancelling 16889826 on provider astro 2024-03-07 10:33:46 proc_lsst.multi:146 [INFO] found job 16889827 in provider astro proc_lsst.multi INFO: found job 16889827 in provider astro 2024-03-07 10:33:46 proc_lsst.multi:201 [INFO] cancelling 16889827 on provider astro proc_lsst.multi INFO: cancelling 16889827 on provider astro 2024-03-07 10:33:46 proc_lsst.multi:146 [INFO] found job 16889828 in provider astro proc_lsst.multi INFO: found job 16889828 in provider astro 2024-03-07 10:33:46 proc_lsst.multi:201 [INFO] cancelling 16889828 on provider astro proc_lsst.multi INFO: cancelling 16889828 on provider astro 2024-03-07 10:33:46 proc_lsst.multi:146 [INFO] found job 16889829 in provider astro proc_lsst.multi INFO: found job 16889829 in provider astro 2024-03-07 10:33:46 proc_lsst.multi:201 [INFO] cancelling 16889829 on provider astro proc_lsst.multi INFO: cancelling 16889829 on provider astro 2024-03-07 10:33:46 proc_lsst.multi:146 [INFO] found job 16889830 in provider astro proc_lsst.multi INFO: found job 16889830 in provider astro 2024-03-07 10:33:46 proc_lsst.multi:201 [INFO] cancelling 16889830 on provider astro proc_lsst.multi INFO: cancelling 16889830 on provider astro 2024-03-07 10:33:46 proc_lsst.multi:146 [INFO] found job 16889831 in provider astro proc_lsst.multi INFO: found job 16889831 in provider astro 2024-03-07 10:33:46 proc_lsst.multi:201 [INFO] cancelling 16889831 on provider astro proc_lsst.multi INFO: cancelling 16889831 on provider astro 2024-03-07 10:33:46 proc_lsst.multi:146 [INFO] found job 16889832 in provider astro proc_lsst.multi INFO: found job 16889832 in provider astro 2024-03-07 10:33:46 proc_lsst.multi:201 [INFO] cancelling 16889832 on provider astro proc_lsst.multi INFO: cancelling 16889832 on provider astro 2024-03-07 10:33:47 proc_lsst.multi:146 [INFO] found job 16889833 in provider astro proc_lsst.multi INFO: found job 16889833 in provider astro 2024-03-07 10:33:47 proc_lsst.multi:201 [INFO] cancelling 16889833 on provider astro proc_lsst.multi INFO: cancelling 16889833 on provider astro 2024-03-07 10:33:47 proc_lsst.multi:146 [INFO] found job 16889834 in provider astro proc_lsst.multi INFO: found job 16889834 in provider astro 2024-03-07 10:33:47 proc_lsst.multi:201 [INFO] cancelling 16889834 on provider astro proc_lsst.multi INFO: cancelling 16889834 on provider astro 2024-03-07 10:33:47 proc_lsst.multi:146 [INFO] found job 16889835 in provider astro proc_lsst.multi INFO: found job 16889835 in provider astro 2024-03-07 10:33:47 proc_lsst.multi:201 [INFO] cancelling 16889835 on provider astro proc_lsst.multi INFO: cancelling 16889835 on provider astro 2024-03-07 10:33:47 proc_lsst.multi:146 [INFO] found job 16889836 in provider astro proc_lsst.multi INFO: found job 16889836 in provider astro 2024-03-07 10:33:47 proc_lsst.multi:201 [INFO] cancelling 16889836 on provider astro proc_lsst.multi INFO: cancelling 16889836 on provider astro 2024-03-07 10:33:47 proc_lsst.multi:146 [INFO] found job 16889837 in provider astro proc_lsst.multi INFO: found job 16889837 in provider astro 2024-03-07 10:33:47 proc_lsst.multi:201 [INFO] cancelling 16889837 on provider astro proc_lsst.multi INFO: cancelling 16889837 on provider astro 2024-03-07 10:33:47 proc_lsst.multi:146 [INFO] found job 16889838 in provider astro proc_lsst.multi INFO: found job 16889838 in provider astro 2024-03-07 10:33:47 proc_lsst.multi:201 [INFO] cancelling 16889838 on provider astro proc_lsst.multi INFO: cancelling 16889838 on provider astro 2024-03-07 10:33:47 proc_lsst.multi:146 [INFO] found job 16889839 in provider astro proc_lsst.multi INFO: found job 16889839 in provider astro 2024-03-07 10:33:47 proc_lsst.multi:201 [INFO] cancelling 16889839 on provider astro proc_lsst.multi INFO: cancelling 16889839 on provider astro 2024-03-07 10:33:48 proc_lsst.multi:146 [INFO] found job 16889840 in provider astro proc_lsst.multi INFO: found job 16889840 in provider astro 2024-03-07 10:33:48 proc_lsst.multi:201 [INFO] cancelling 16889840 on provider astro proc_lsst.multi INFO: cancelling 16889840 on provider astro 2024-03-07 10:33:48 proc_lsst.multi:146 [INFO] found job 16889841 in provider astro proc_lsst.multi INFO: found job 16889841 in provider astro 2024-03-07 10:33:48 proc_lsst.multi:201 [INFO] cancelling 16889841 on provider astro proc_lsst.multi INFO: cancelling 16889841 on provider astro 2024-03-07 10:33:48 proc_lsst.multi:146 [INFO] found job 16889842 in provider astro proc_lsst.multi INFO: found job 16889842 in provider astro 2024-03-07 10:33:48 proc_lsst.multi:201 [INFO] cancelling 16889842 on provider astro proc_lsst.multi INFO: cancelling 16889842 on provider astro 2024-03-07 10:33:48 proc_lsst.multi:146 [INFO] found job 16889843 in provider astro proc_lsst.multi INFO: found job 16889843 in provider astro 2024-03-07 10:33:48 proc_lsst.multi:201 [INFO] cancelling 16889843 on provider astro proc_lsst.multi INFO: cancelling 16889843 on provider astro 2024-03-07 10:33:48 proc_lsst.multi:146 [INFO] found job 16889844 in provider astro proc_lsst.multi INFO: found job 16889844 in provider astro 2024-03-07 10:33:48 proc_lsst.multi:201 [INFO] cancelling 16889844 on provider astro proc_lsst.multi INFO: cancelling 16889844 on provider astro 2024-03-07 10:33:48 proc_lsst.multi:146 [INFO] found job 16889845 in provider astro proc_lsst.multi INFO: found job 16889845 in provider astro 2024-03-07 10:33:48 proc_lsst.multi:201 [INFO] cancelling 16889845 on provider astro proc_lsst.multi INFO: cancelling 16889845 on provider astro 2024-03-07 10:33:48 proc_lsst.multi:146 [INFO] found job 16889846 in provider astro proc_lsst.multi INFO: found job 16889846 in provider astro 2024-03-07 10:33:48 proc_lsst.multi:201 [INFO] cancelling 16889846 on provider astro proc_lsst.multi INFO: cancelling 16889846 on provider astro 2024-03-07 10:33:48 proc_lsst.multi:146 [INFO] found job 16889847 in provider astro proc_lsst.multi INFO: found job 16889847 in provider astro 2024-03-07 10:33:48 proc_lsst.multi:201 [INFO] cancelling 16889847 on provider astro proc_lsst.multi INFO: cancelling 16889847 on provider astro 2024-03-07 10:33:48 proc_lsst.multi:146 [INFO] found job 16889848 in provider astro proc_lsst.multi INFO: found job 16889848 in provider astro 2024-03-07 10:33:48 proc_lsst.multi:201 [INFO] cancelling 16889848 on provider astro proc_lsst.multi INFO: cancelling 16889848 on provider astro 2024-03-07 10:33:49 proc_lsst.multi:146 [INFO] found job 16889852 in provider astro proc_lsst.multi INFO: found job 16889852 in provider astro 2024-03-07 10:33:49 proc_lsst.multi:201 [INFO] cancelling 16889852 on provider astro proc_lsst.multi INFO: cancelling 16889852 on provider astro 2024-03-07 10:33:49 proc_lsst.multi:146 [INFO] found job 16889853 in provider astro proc_lsst.multi INFO: found job 16889853 in provider astro 2024-03-07 10:33:49 proc_lsst.multi:201 [INFO] cancelling 16889853 on provider astro proc_lsst.multi INFO: cancelling 16889853 on provider astro 2024-03-07 10:33:49 proc_lsst.multi:146 [INFO] found job 16889854 in provider astro proc_lsst.multi INFO: found job 16889854 in provider astro 2024-03-07 10:33:49 proc_lsst.multi:201 [INFO] cancelling 16889854 on provider astro proc_lsst.multi INFO: cancelling 16889854 on provider astro 2024-03-07 10:33:49 proc_lsst.multi:146 [INFO] found job 16889855 in provider astro proc_lsst.multi INFO: found job 16889855 in provider astro 2024-03-07 10:33:49 proc_lsst.multi:201 [INFO] cancelling 16889855 on provider astro proc_lsst.multi INFO: cancelling 16889855 on provider astro 2024-03-07 10:33:49 proc_lsst.multi:146 [INFO] found job 16889861 in provider escience_a40 proc_lsst.multi INFO: found job 16889861 in provider escience_a40 2024-03-07 10:33:49 proc_lsst.multi:201 [INFO] cancelling 16889861 on provider escience_a40 proc_lsst.multi INFO: cancelling 16889861 on provider escience_a40 2024-03-07 10:33:49 proc_lsst.multi:146 [INFO] found job 16889862 in provider escience_a40 proc_lsst.multi INFO: found job 16889862 in provider escience_a40 2024-03-07 10:33:49 proc_lsst.multi:201 [INFO] cancelling 16889862 on provider escience_a40 proc_lsst.multi INFO: cancelling 16889862 on provider escience_a40 2024-03-07 10:33:49 proc_lsst.multi:146 [INFO] found job 16889863 in provider escience_a40 proc_lsst.multi INFO: found job 16889863 in provider escience_a40 2024-03-07 10:33:49 proc_lsst.multi:201 [INFO] cancelling 16889863 on provider escience_a40 proc_lsst.multi INFO: cancelling 16889863 on provider escience_a40 2024-03-07 10:33:49 proc_lsst.multi:146 [INFO] found job 16889864 in provider escience_a40 proc_lsst.multi INFO: found job 16889864 in provider escience_a40 2024-03-07 10:33:49 proc_lsst.multi:201 [INFO] cancelling 16889864 on provider escience_a40 proc_lsst.multi INFO: cancelling 16889864 on provider escience_a40 2024-03-07 10:33:49 proc_lsst.multi:146 [INFO] found job 16889865 in provider escience_a40 proc_lsst.multi INFO: found job 16889865 in provider escience_a40 2024-03-07 10:33:49 proc_lsst.multi:201 [INFO] cancelling 16889865 on provider escience_a40 proc_lsst.multi INFO: cancelling 16889865 on provider escience_a40 2024-03-07 10:33:49 proc_lsst.multi:146 [INFO] found job 16889866 in provider escience_a40 proc_lsst.multi INFO: found job 16889866 in provider escience_a40 2024-03-07 10:33:49 proc_lsst.multi:201 [INFO] cancelling 16889866 on provider escience_a40 proc_lsst.multi INFO: cancelling 16889866 on provider escience_a40 2024-03-07 10:33:49 proc_lsst.multi:146 [INFO] found job 16889867 in provider escience_a40 proc_lsst.multi INFO: found job 16889867 in provider escience_a40 2024-03-07 10:33:49 proc_lsst.multi:201 [INFO] cancelling 16889867 on provider escience_a40 proc_lsst.multi INFO: cancelling 16889867 on provider escience_a40 2024-03-07 10:33:50 proc_lsst.multi:146 [INFO] found job 16889868 in provider escience_a40 proc_lsst.multi INFO: found job 16889868 in provider escience_a40 2024-03-07 10:33:50 proc_lsst.multi:201 [INFO] cancelling 16889868 on provider escience_a40 proc_lsst.multi INFO: cancelling 16889868 on provider escience_a40 2024-03-07 10:33:50 proc_lsst.multi:146 [INFO] found job 16889869 in provider escience_a40 proc_lsst.multi INFO: found job 16889869 in provider escience_a40 2024-03-07 10:33:50 proc_lsst.multi:201 [INFO] cancelling 16889869 on provider escience_a40 proc_lsst.multi INFO: cancelling 16889869 on provider escience_a40 2024-03-07 10:33:50 proc_lsst.multi:146 [INFO] found job 16889870 in provider escience_a40 proc_lsst.multi INFO: found job 16889870 in provider escience_a40 2024-03-07 10:33:50 proc_lsst.multi:201 [INFO] cancelling 16889870 on provider escience_a40 proc_lsst.multi INFO: cancelling 16889870 on provider escience_a40 2024-03-07 10:33:50 proc_lsst.multi:146 [INFO] found job 16889871 in provider escience_a40 proc_lsst.multi INFO: found job 16889871 in provider escience_a40 2024-03-07 10:33:50 proc_lsst.multi:201 [INFO] cancelling 16889871 on provider escience_a40 proc_lsst.multi INFO: cancelling 16889871 on provider escience_a40 2024-03-07 10:33:50 proc_lsst.multi:146 [INFO] found job 16889856 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889856 in provider escience_rtx6k 2024-03-07 10:33:50 proc_lsst.multi:201 [INFO] cancelling 16889856 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16889856 on provider escience_rtx6k 2024-03-07 10:33:50 proc_lsst.multi:146 [INFO] found job 16889857 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889857 in provider escience_rtx6k 2024-03-07 10:33:50 proc_lsst.multi:201 [INFO] cancelling 16889857 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16889857 on provider escience_rtx6k 2024-03-07 10:33:50 proc_lsst.multi:146 [INFO] found job 16889858 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889858 in provider escience_rtx6k 2024-03-07 10:33:50 proc_lsst.multi:201 [INFO] cancelling 16889858 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16889858 on provider escience_rtx6k 2024-03-07 10:33:50 proc_lsst.multi:146 [INFO] found job 16889859 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889859 in provider escience_rtx6k 2024-03-07 10:33:50 proc_lsst.multi:201 [INFO] cancelling 16889859 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16889859 on provider escience_rtx6k 2024-03-07 10:33:50 proc_lsst.multi:146 [INFO] found job 16889860 in provider escience_rtx6k proc_lsst.multi INFO: found job 16889860 in provider escience_rtx6k 2024-03-07 10:33:50 proc_lsst.multi:201 [INFO] cancelling 16889860 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16889860 on provider escience_rtx6k 2024-03-07 10:33:50 proc_lsst.multi:146 [INFO] found job 24403 in provider local proc_lsst.multi INFO: found job 24403 in provider local 2024-03-07 10:33:50 proc_lsst.multi:201 [INFO] cancelling 24403 on provider local proc_lsst.multi INFO: cancelling 24403 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 24403 parsl.providers.local.local WARNING: Failed to kill PID: 24403 and child processes on local 2024-03-07 10:33:51 proc_lsst.multi:50 [INFO] no new jobs since last cancel, resuming executor shutdown proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt parsl.dataflow.dflow INFO: Shut down executor multi parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete parsl.executors.threads DEBUG: Done with executor shutdown parsl.dataflow.dflow INFO: Shut down executor _parsl_internal parsl.dataflow.dflow INFO: Terminated executors parsl.dataflow.dflow INFO: DFK cleanup complete parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 687.1684 seconds lsst.ctrl.bps.drivers INFO: Run 'DEEP_20220825_B1b_science#step1_20240307T102204Z' submitted for execution with id 'None' lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 687.1915 seconds; current memory usage: 0.370 Gibyte, delta: 0.006 Gibyte, peak delta: 0.006 Gibyte lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 705.1867 seconds; current memory usage: 0.370 Gibyte, delta: 0.233 Gibyte, peak delta: 0.233 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.370 Gibyte (main), 0.370 Gibyte (largest child process) Run Id: None Run Name: DEEP_20220825_B1b_science#step1_20240307T102204Z parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up