+ PROC_LSST_SUBMIT_PATH=/mmfs1/home/stevengs/dirac/DEEP/submit + bps submit /gscratch/dirac/shared/opt/proc_lsst/pipelines/submit.yaml -b /mmfs1/home/stevengs/dirac/DEEP/repo -i DEEP/20190926/B1a --output-run DEEP/20190926/B1a/science#step1/20240313T084347Z --qgraph pipeline.qgraph lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process. lsst.ctrl.bps.drivers INFO: Starting submission process lsst.ctrl.bps.drivers INFO: Initializing execution environment lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 3.0686 seconds; current memory usage: 0.180 Gibyte, delta: 0.042 Gibyte, peak delta: 0.054 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.193 Gibyte (main), 0.000 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph) lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.0040 seconds lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.0164 seconds lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/pipeline_orig.qgraph DEEP/20190926/B1a/science#step1/20240313T084347Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/pipeline.qgraph lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 4.2488 seconds lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 10.5262 seconds; current memory usage: 0.344 Gibyte, delta: 0.164 Gibyte, peak delta: 0.151 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.344 Gibyte (main), 0.336 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs) lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0109 seconds; current memory usage: 0.344 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.344 Gibyte (main), 0.336 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 250 cluster(s) lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow) lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20190926_B1a_science#step1_20240313T084347Z' lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 0.0622 seconds; current memory usage: 0.344 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.344 Gibyte (main), 0.336 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 252 job(s) (including final) lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow) parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2 Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses s_addresses.add(address_by_interface(interface)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper retval = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface return socket.inet_ntoa(fcntl.ioctl( ^^^^^^^^^^^^ OSError: [Errno 99] Cannot assign requested address lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 0.1967 seconds; current memory usage: 0.345 Gibyte, delta: 0.001 Gibyte, peak delta: 0.001 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.345 Gibyte (main), 0.336 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting submit stage lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000 parsl.dataflow.dflow INFO: Starting DataFlowKernel with config Config( app_cache=True, checkpoint_files=None, checkpoint_mode='task_exit', checkpoint_period=None, executors=[MultiHighThroughputExecutor()], garbage_collect=True, initialize_logging=True, internal_tasks_max_threads=10, max_idletime=120.0, monitoring=None, retries=1, retry_handler=None, run_dir='runinfo', strategy='simple', usage_tracking=False ) parsl.dataflow.dflow INFO: Parsl version: 2023.06.12 parsl.usage_tracking.usage DEBUG: Tracking status: False parsl.dataflow.dflow INFO: Run id is: 4cf232fa-0c3f-461f-b9e6-fe2f67762499 parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name parsl.dataflow.memoization INFO: App caching initialized parsl.dataflow.strategy DEBUG: Scaling strategy: simple parsl.executors.high_throughput.executor DEBUG: Starting queue management thread parsl.executors.high_throughput.executor DEBUG: queue management worker starting parsl.executors.high_throughput.executor DEBUG: Started queue management thread Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z parsl.executors.high_throughput.executor DEBUG: Created management thread: parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id={block_id} --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider: parsl.executors.status_handling INFO: Scaling out by 1 blocks parsl.executors.status_handling INFO: Allocated block ID 0 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0 2024-03-13 08:44:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 2024-03-13 08:44:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 08:44:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-13 08:44:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 08:44:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local parsl.providers.local.local DEBUG: Launching in remote mode 2024-03-13 08:44:50 proc_lsst.multi:170 [INFO] [multi] job_id 34092 proc_lsst.multi INFO: [multi] job_id 34092 2024-03-13 08:44:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:44:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 08:44:50 proc_lsst.multi:178 [INFO] [multi] provider local accepted submit and returned 34092 proc_lsst.multi INFO: [multi] provider local accepted submit and returned 34092 parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 34092 parsl.dataflow.job_status_poller DEBUG: Adding executor multi parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 0 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 0 has memoization hash 09066a0219e24fdbb84bb3def6e26179 parsl.dataflow.memoization INFO: Task 0 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b682c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1 parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898348/81898b87-3975-4957-9c7a-c96ad661712e_calibrate_898348_31.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898348/81898b87-3975-4957-9c7a-c96ad661712e_calibrate_898348_31.stderr parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1 has memoization hash 66ab7b72d091b964779f664b0e32bb80 parsl.dataflow.memoization INFO: Task 1 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b68680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 2 parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898344/5582ca13-75db-4a41-ab17-540aa0a69df1_calibrate_898344_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898344/5582ca13-75db-4a41-ab17-540aa0a69df1_calibrate_898344_31.stderr parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2 has memoization hash 26209eddfa42689388c6f4e0bfdc4f38 parsl.dataflow.memoization INFO: Task 2 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b687c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 3 parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898333/6df343bb-eaa6-4a48-a6cd-e1dd2bbc6bb7_calibrate_898333_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898333/6df343bb-eaa6-4a48-a6cd-e1dd2bbc6bb7_calibrate_898333_61.stderr parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3 has memoization hash ac45f7da037ff0da36555fa281bb6ff8 parsl.dataflow.memoization INFO: Task 3 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b689a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 4 parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898350/ac90d1dd-77a7-4e35-9414-e727205aec56_calibrate_898350_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898350/ac90d1dd-77a7-4e35-9414-e727205aec56_calibrate_898350_31.stderr parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash 1b63349bb77c9e57be695c546ea22330 parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b68a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 5 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898375/21d6a845-9342-4d4b-907e-e59a7da8b7d0_calibrate_898375_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898375/21d6a845-9342-4d4b-907e-e59a7da8b7d0_calibrate_898375_61.stderr parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5 has memoization hash 7ab3a43ba4690db27a9d2acbf2b23156 parsl.dataflow.memoization INFO: Task 5 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b68ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 6 parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898326/e8dc36e3-899c-4e77-95b7-12cea144d25c_calibrate_898326_31.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898326/e8dc36e3-899c-4e77-95b7-12cea144d25c_calibrate_898326_31.stderr parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 6 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 6 has memoization hash d97d0a7bdaf10f72d049ce95d7d29033 parsl.dataflow.memoization INFO: Task 6 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3a3e1b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 7 parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898374/0eeb0a0c-dc47-435f-bb6a-652c42051f5c_calibrate_898374_61.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898374/0eeb0a0c-dc47-435f-bb6a-652c42051f5c_calibrate_898374_61.stderr parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 7 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 7 has memoization hash 7e4a79586e0a0d69281374a58e04e7d9 parsl.dataflow.memoization INFO: Task 7 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b685e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 8 parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898342/dc2c0f58-ade0-4f10-9908-3da1526301af_calibrate_898342_31.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898342/dc2c0f58-ade0-4f10-9908-3da1526301af_calibrate_898342_31.stderr parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 8 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 8 has memoization hash 0ea846d7f41dde9c5463dfa6179c7af3 parsl.dataflow.memoization INFO: Task 8 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b68cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 9 parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898366/3faa2151-682e-4b26-b2ce-a961451dc732_calibrate_898366_61.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898366/3faa2151-682e-4b26-b2ce-a961451dc732_calibrate_898366_61.stderr parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 9 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 9 has memoization hash 6bdcb389812a683b49c783b6c448d66b parsl.dataflow.memoization INFO: Task 9 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b68400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 10 parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898382/4ab7025a-27b7-4b84-91f5-ad14420c245e_calibrate_898382_7.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898382/4ab7025a-27b7-4b84-91f5-ad14420c245e_calibrate_898382_7.stderr parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 10 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash b5cabc075ea894567234827bf4b39aaa parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b68900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 11 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898336/39bafb1f-6aae-4e48-9cf8-9bfc2976a642_calibrate_898336_61.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898336/39bafb1f-6aae-4e48-9cf8-9bfc2976a642_calibrate_898336_61.stderr parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 11 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 11 has memoization hash a02de1cca403ca71bfd1ce8526d5b68a parsl.dataflow.memoization INFO: Task 11 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b68c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 12 parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898342/cf783a12-8612-47c1-ad91-69a33e09a85c_calibrate_898342_61.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898342/cf783a12-8612-47c1-ad91-69a33e09a85c_calibrate_898342_61.stderr parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 12 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 12 has memoization hash cad4018ac34f8c2db58928cbebfbe4e7 parsl.dataflow.memoization INFO: Task 12 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b68f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 13 parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898349/353d01b1-13b9-47ef-8df8-b3c7f7d84717_calibrate_898349_31.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898349/353d01b1-13b9-47ef-8df8-b3c7f7d84717_calibrate_898349_31.stderr parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 13 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 13 has memoization hash c72cc05cbe5d0f3f46a35f41069e2993 parsl.dataflow.memoization INFO: Task 13 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b69120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 14 parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898377/769a2947-4530-42bc-9032-1647e8b4baa3_calibrate_898377_31.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898377/769a2947-4530-42bc-9032-1647e8b4baa3_calibrate_898377_31.stderr parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 14 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 14 has memoization hash 976b0c75f578f168831d93d747f77b70 parsl.dataflow.memoization INFO: Task 14 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b69300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 15 parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898356/1070cfd5-8ce6-41fc-a039-2d4ac71300c0_calibrate_898356_61.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898356/1070cfd5-8ce6-41fc-a039-2d4ac71300c0_calibrate_898356_61.stderr parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 15 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 15 has memoization hash 4302c68bf760c49455c325a400d545a9 parsl.dataflow.memoization INFO: Task 15 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b68ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 16 parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898341/b35cd10a-e0ae-4f4d-bded-3b07f4776216_calibrate_898341_31.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898341/b35cd10a-e0ae-4f4d-bded-3b07f4776216_calibrate_898341_31.stderr parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 16 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash 4b3acee8a57858b7460670ddb218b574 parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b693a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 17 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898363/9a3d4b9b-637b-4d54-9756-08db927791ea_calibrate_898363_61.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898363/9a3d4b9b-637b-4d54-9756-08db927791ea_calibrate_898363_61.stderr parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 17 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 17 has memoization hash c3be2f6c16e1ce750d1a80c216b86dbf parsl.dataflow.memoization INFO: Task 17 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b69580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 18 parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898369/53ab6566-4b30-4473-a020-ede08ff4c708_calibrate_898369_61.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898369/53ab6566-4b30-4473-a020-ede08ff4c708_calibrate_898369_61.stderr parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 18 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 18 has memoization hash 7b65f5e5c021803b62472089700ad251 parsl.dataflow.memoization INFO: Task 18 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b69260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 19 parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898362/d41fc6e8-b2b1-467f-a720-63f17e62b88c_calibrate_898362_61.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898362/d41fc6e8-b2b1-467f-a720-63f17e62b88c_calibrate_898362_61.stderr parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 19 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 19 has memoization hash 2dd5cc0de2690a25ddf96527927bdc84 parsl.dataflow.memoization INFO: Task 19 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b69440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 20 parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898335/9ee28e39-dd1f-40cc-91e9-458e31ad9e35_calibrate_898335_61.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898335/9ee28e39-dd1f-40cc-91e9-458e31ad9e35_calibrate_898335_61.stderr parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 20 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 20 has memoization hash 68bf16d6705930d3552ef85355e36440 parsl.dataflow.memoization INFO: Task 20 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b696c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 21 parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898346/fa1671bf-7a1a-4e67-8fbe-bb109b12b349_calibrate_898346_31.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898346/fa1671bf-7a1a-4e67-8fbe-bb109b12b349_calibrate_898346_31.stderr parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 21 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 21 has memoization hash 3c22e053a9a4ae24e4b55a94b408d45f parsl.dataflow.memoization INFO: Task 21 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b69620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 22 parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898353/c49c449d-4f7a-4ba2-91fa-945b7c653983_calibrate_898353_31.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898353/c49c449d-4f7a-4ba2-91fa-945b7c653983_calibrate_898353_31.stderr parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 22 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash 483dcf17c6bb676a0286f433a5d209d2 parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b69800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 23 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898366/78353f3c-57e1-4237-8cc1-01f5ac43e0ee_calibrate_898366_31.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898366/78353f3c-57e1-4237-8cc1-01f5ac43e0ee_calibrate_898366_31.stderr parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 23 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 23 has memoization hash 8144802c995d4528edd2552745882d99 parsl.dataflow.memoization INFO: Task 23 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b69940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 24 parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898345/5f136065-162d-49e3-9de7-31bf464a799c_calibrate_898345_61.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898345/5f136065-162d-49e3-9de7-31bf464a799c_calibrate_898345_61.stderr parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 24 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 24 has memoization hash 4a0f9e7707a025e86a9bce98ffc50f66 parsl.dataflow.memoization INFO: Task 24 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b69760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 25 parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898331/5c785658-adcd-419d-b339-11f13f7ccddd_calibrate_898331_61.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898331/5c785658-adcd-419d-b339-11f13f7ccddd_calibrate_898331_61.stderr parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 25 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 25 has memoization hash 24efdbd9c833bcea3669feca1f99e441 parsl.dataflow.memoization INFO: Task 25 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b699e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 26 parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898319/61241879-2ac8-44d1-9098-dca5bb7dd64e_calibrate_898319_61.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898319/61241879-2ac8-44d1-9098-dca5bb7dd64e_calibrate_898319_61.stderr parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 26 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 26 has memoization hash 404a922ad98363cbbcd02ddc8c89da0a parsl.dataflow.memoization INFO: Task 26 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b698a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 27 parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898374/b2ab386a-7074-4980-a0ae-626e602344c7_calibrate_898374_31.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898374/b2ab386a-7074-4980-a0ae-626e602344c7_calibrate_898374_31.stderr parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 27 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 27 has memoization hash 2154875a4ca8ccfdcf0b441f8ebe8127 parsl.dataflow.memoization INFO: Task 27 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b69b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 28 parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898334/38bdbda3-4974-4bd8-b77c-ac6f04c45a29_calibrate_898334_31.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898334/38bdbda3-4974-4bd8-b77c-ac6f04c45a29_calibrate_898334_31.stderr parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 28 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash 9da04a8eed0e599dcf293179a3c308c4 parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b69c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 29 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898375/48301fc1-2890-4842-b21b-b4bda43648de_calibrate_898375_31.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898375/48301fc1-2890-4842-b21b-b4bda43648de_calibrate_898375_31.stderr parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 29 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 29 has memoization hash 4039dedca7abdad7bd0d0159c6a786f7 parsl.dataflow.memoization INFO: Task 29 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b69bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 29 try 0 launched on executor multi with executor id 30 parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898320/4d762e6c-fe69-40c5-a31f-25c11aca98cb_calibrate_898320_31.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898320/4d762e6c-fe69-40c5-a31f-25c11aca98cb_calibrate_898320_31.stderr parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 30 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 30 has memoization hash 6fd2140ec551b4a45f963c0c020e2ebf parsl.dataflow.memoization INFO: Task 30 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b69a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 31 parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898334/b995a848-ee4d-4405-a8c0-dda0691b96b0_calibrate_898334_61.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898334/b995a848-ee4d-4405-a8c0-dda0691b96b0_calibrate_898334_61.stderr parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 31 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 31 has memoization hash bcfe0a168ab99bf456c8b1f0cdfa7d07 parsl.dataflow.memoization INFO: Task 31 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b69da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 32 parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898343/64d0e1db-ccf1-458e-bbd9-d102c90d7e8b_calibrate_898343_61.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898343/64d0e1db-ccf1-458e-bbd9-d102c90d7e8b_calibrate_898343_61.stderr parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 32 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 32 has memoization hash ad83b57306beb23bf2701b4ec8428fe5 parsl.dataflow.memoization INFO: Task 32 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b69e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 33 parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898351/f068b0a2-eb89-4a63-b361-e8d1e72e53c7_calibrate_898351_31.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898351/f068b0a2-eb89-4a63-b361-e8d1e72e53c7_calibrate_898351_31.stderr parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 33 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 33 has memoization hash c88a06f6a83e0593b438fb1774dd2460 parsl.dataflow.memoization INFO: Task 33 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b69d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 34 parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898344/bf5bb1bb-5ab6-47f9-8e07-57808bd6e728_calibrate_898344_61.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898344/bf5bb1bb-5ab6-47f9-8e07-57808bd6e728_calibrate_898344_61.stderr parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 34 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash 5947a47f671ed559925e25b77189cefd parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b69ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 35 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898354/c244ec8e-369a-448b-a0ee-6e385648f4b1_calibrate_898354_31.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898354/c244ec8e-369a-448b-a0ee-6e385648f4b1_calibrate_898354_31.stderr parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 35 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 35 has memoization hash f7549c3b5ee5ec0cd39d9e522a784dd2 parsl.dataflow.memoization INFO: Task 35 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b69f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 36 parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898307/a34da88c-ce76-49a3-9a04-93e927f2d5fa_calibrate_898307_31.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898307/a34da88c-ce76-49a3-9a04-93e927f2d5fa_calibrate_898307_31.stderr parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 36 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 36 has memoization hash 61aee1dbd6fba070a7190edb6496165b parsl.dataflow.memoization INFO: Task 36 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 37 parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898309/82e1efa9-7097-42b4-aede-e23e05ddbd81_calibrate_898309_61.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898309/82e1efa9-7097-42b4-aede-e23e05ddbd81_calibrate_898309_61.stderr parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 37 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 37 has memoization hash 9bbeccbae9becdd64a681642865bf01b parsl.dataflow.memoization INFO: Task 37 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 38 parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898329/0ba2197b-c2e4-4473-a0f4-3c1dedb841c1_calibrate_898329_61.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898329/0ba2197b-c2e4-4473-a0f4-3c1dedb841c1_calibrate_898329_61.stderr parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 38 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 38 has memoization hash 2cd84899ae596971a5725f38d92779eb parsl.dataflow.memoization INFO: Task 38 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 39 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898337/44b1f232-8e71-4f34-8980-bb9b465c63b1_calibrate_898337_61.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898337/44b1f232-8e71-4f34-8980-bb9b465c63b1_calibrate_898337_61.stderr parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 39 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 39 has memoization hash 5d36b7e58d4a9ec85cab8d9824bd8fee parsl.dataflow.memoization INFO: Task 39 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 40 parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898341/22b89990-f444-407a-a373-03704d655796_calibrate_898341_61.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898341/22b89990-f444-407a-a373-03704d655796_calibrate_898341_61.stderr parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 40 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash c7f9ef97f36aad0c3ad3f136aee93c45 parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 41 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898334/39685529-733b-419f-b862-f7a086c8ce6a_calibrate_898334_36.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898334/39685529-733b-419f-b862-f7a086c8ce6a_calibrate_898334_36.stderr parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 41 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 41 has memoization hash 9ca2ae210d2c757b4188bbf1deab3a05 parsl.dataflow.memoization INFO: Task 41 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 42 parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898324/8d56e6fa-7a2c-4b9b-afa3-e24d76c7887b_calibrate_898324_61.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898324/8d56e6fa-7a2c-4b9b-afa3-e24d76c7887b_calibrate_898324_61.stderr parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 42 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 42 has memoization hash c4e1e65f38d0257accd555cc0a75ca7b parsl.dataflow.memoization INFO: Task 42 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6b2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 43 parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898312/5cfe0bcb-2d52-4157-b2cd-bd5b702680b3_calibrate_898312_31.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898312/5cfe0bcb-2d52-4157-b2cd-bd5b702680b3_calibrate_898312_31.stderr parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 43 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 43 has memoization hash 68e95a36b0438dbfc00da1206a9ce4d5 parsl.dataflow.memoization INFO: Task 43 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 44 parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898336/f5525908-304f-433e-9c72-7dddccdbfb5c_calibrate_898336_31.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898336/f5525908-304f-433e-9c72-7dddccdbfb5c_calibrate_898336_31.stderr parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 44 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 44 has memoization hash 4069964c40da3c18bb04d283d4cd2a2e parsl.dataflow.memoization INFO: Task 44 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 45 parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898324/f7963f80-1c9c-4bb7-97aa-466c1c22bdb9_calibrate_898324_31.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898324/f7963f80-1c9c-4bb7-97aa-466c1c22bdb9_calibrate_898324_31.stderr parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 45 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 45 has memoization hash 98fd1d3c7031e1dbd8391f88c1a617db parsl.dataflow.memoization INFO: Task 45 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 46 parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898322/917bf878-02ae-462d-8477-3109632be425_calibrate_898322_61.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898322/917bf878-02ae-462d-8477-3109632be425_calibrate_898322_61.stderr parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 46 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 46 has memoization hash 8f8e36e3614cf9d2377fccbbc80d45b4 parsl.dataflow.memoization INFO: Task 46 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 47 parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898332/1afe740a-c14b-4016-ba53-e7fbd2c9f2fc_calibrate_898332_61.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898332/1afe740a-c14b-4016-ba53-e7fbd2c9f2fc_calibrate_898332_61.stderr parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 47 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 47 has memoization hash 495796d6dd275cdebea5a7947f2db62b parsl.dataflow.memoization INFO: Task 47 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6afc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 48 parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898365/68f3688a-de27-4713-83e4-4a8332c2e510_calibrate_898365_61.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898365/68f3688a-de27-4713-83e4-4a8332c2e510_calibrate_898365_61.stderr parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 48 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 48 has memoization hash 81651c4a5cc7c31adf702f2a62e10b61 parsl.dataflow.memoization INFO: Task 48 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 49 parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898382/26d7638c-432a-48eb-8a5e-2bd420831861_calibrate_898382_61.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898382/26d7638c-432a-48eb-8a5e-2bd420831861_calibrate_898382_61.stderr parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 49 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 49 has memoization hash 61298ebe490fd5b5bcb32fbd1ea462ad parsl.dataflow.memoization INFO: Task 49 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 50 parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898327/f0e8c5c2-1e4e-4b94-97cf-0e192997e8f0_calibrate_898327_31.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898327/f0e8c5c2-1e4e-4b94-97cf-0e192997e8f0_calibrate_898327_31.stderr parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 50 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 50 has memoization hash 362734b5a790c00cc0b62964f63e77c9 parsl.dataflow.memoization INFO: Task 50 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 51 parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898323/93fbc6ff-c819-4d23-b892-475cc0988494_calibrate_898323_61.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898323/93fbc6ff-c819-4d23-b892-475cc0988494_calibrate_898323_61.stderr parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 51 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 51 has memoization hash 0a6b63a644cafb3f45916f28e2dcab00 parsl.dataflow.memoization INFO: Task 51 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 52 parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898353/b15df392-ec28-447e-81c3-72d987d085c6_calibrate_898353_61.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898353/b15df392-ec28-447e-81c3-72d987d085c6_calibrate_898353_61.stderr parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 52 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash 546bcd619bfed86dbc58546373cc23c9 parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 53 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898352/3a68dde5-3364-44e6-9bda-ed56a9f9eaac_calibrate_898352_31.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898352/3a68dde5-3364-44e6-9bda-ed56a9f9eaac_calibrate_898352_31.stderr parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 53 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 53 has memoization hash 5643377b5f14ba72e85f5850cb6406a2 parsl.dataflow.memoization INFO: Task 53 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 54 parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898346/148e6dc2-a926-488e-8cfb-2a923573e179_calibrate_898346_61.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898346/148e6dc2-a926-488e-8cfb-2a923573e179_calibrate_898346_61.stderr parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 54 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 54 has memoization hash 71a530e2f28b0764e94bcbd4c6ecbd86 parsl.dataflow.memoization INFO: Task 54 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 55 parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898318/b13456ab-f9fa-4fb6-81b1-66a42b8b3c13_calibrate_898318_31.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898318/b13456ab-f9fa-4fb6-81b1-66a42b8b3c13_calibrate_898318_31.stderr parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 55 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 55 has memoization hash e093fa6e2ddd0c7d3c73293fa8d0a1cf parsl.dataflow.memoization INFO: Task 55 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 56 parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898331/9940692a-d823-4c18-bb68-1fb4b229ad0f_calibrate_898331_31.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898331/9940692a-d823-4c18-bb68-1fb4b229ad0f_calibrate_898331_31.stderr parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 56 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 56 has memoization hash 8a5856ee7d0c61a4546900cabfbfe1c9 parsl.dataflow.memoization INFO: Task 56 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b694e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 57 parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898301/8ac197d9-bd72-461c-90c7-d994ec4e890c_characterizeImage_898301_2.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898301/8ac197d9-bd72-461c-90c7-d994ec4e890c_characterizeImage_898301_2.stderr parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 57 submitted for App calibrate, waiting on task 56 parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 57 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 58 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 58 has memoization hash 871f1c0bfc2b03edddb157476d0651fe parsl.dataflow.memoization INFO: Task 58 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 58 parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898311/72116209-a5e5-4f23-a885-45dbd5e314e6_characterizeImage_898311_2.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898311/72116209-a5e5-4f23-a885-45dbd5e314e6_characterizeImage_898311_2.stderr parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 59 submitted for App calibrate, waiting on task 58 parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 59 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 60 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 60 has memoization hash 97a9946e5ae719d5aaff1e8b4d89c2e8 parsl.dataflow.memoization INFO: Task 60 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 59 parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898290/ec58caa0-fe40-46e5-a79d-79858acbf8bc_characterizeImage_898290_2.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898290/ec58caa0-fe40-46e5-a79d-79858acbf8bc_characterizeImage_898290_2.stderr parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 61 submitted for App calibrate, waiting on task 60 parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 61 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 62 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 62 has memoization hash 2aba5b874159f2b9bd7bb6adc9fa55da parsl.dataflow.memoization INFO: Task 62 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 60 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898306/687414c1-d1e9-4f0b-9b67-b80b325c7bb7_characterizeImage_898306_2.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898306/687414c1-d1e9-4f0b-9b67-b80b325c7bb7_characterizeImage_898306_2.stderr parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 63 submitted for App calibrate, waiting on task 62 parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 63 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 64 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash cb65e91f672f3957fd9fc5594b209a2b parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 61 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898380/90132901-dc69-42a9-86b3-bcd34dd462c2_characterizeImage_898380_2.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898380/90132901-dc69-42a9-86b3-bcd34dd462c2_characterizeImage_898380_2.stderr parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 65 submitted for App calibrate, waiting on task 64 parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 65 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 66 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 66 has memoization hash 1c3e7bb0f533db41b0d33e340d0bfcc4 parsl.dataflow.memoization INFO: Task 66 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 62 parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898355/10771c4c-8a1e-4e1e-9245-87f11f5ccd3e_characterizeImage_898355_2.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898355/10771c4c-8a1e-4e1e-9245-87f11f5ccd3e_characterizeImage_898355_2.stderr parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 67 submitted for App calibrate, waiting on task 66 parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 67 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 68 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 68 has memoization hash 25e7d311cd1cde0309d3624f17af3185 parsl.dataflow.memoization INFO: Task 68 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 63 parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898334/cfea5b02-a172-4f43-a059-3065b6a16fe7_characterizeImage_898334_2.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898334/cfea5b02-a172-4f43-a059-3065b6a16fe7_characterizeImage_898334_2.stderr parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 69 submitted for App calibrate, waiting on task 68 parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 69 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 70 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash 3b2ca9a94f256047962755948959ef06 parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 64 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898288/256364ee-b666-43fa-9aae-b3d74aa5385a_characterizeImage_898288_2.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898288/256364ee-b666-43fa-9aae-b3d74aa5385a_characterizeImage_898288_2.stderr parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 71 submitted for App calibrate, waiting on task 70 parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 71 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 72 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 72 has memoization hash 5e1fcaaa00a419d8d4e76a5c2bd888c7 parsl.dataflow.memoization INFO: Task 72 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 65 parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898375/9bad3368-9d28-420c-bd65-b5390f0dd3b5_characterizeImage_898375_2.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898375/9bad3368-9d28-420c-bd65-b5390f0dd3b5_characterizeImage_898375_2.stderr parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 73 submitted for App calibrate, waiting on task 72 parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 73 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 74 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 74 has memoization hash 411d81f3ab7f15dad4bf228ec2a2a01a parsl.dataflow.memoization INFO: Task 74 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 66 parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898326/7d3152b1-6692-4435-9366-c053e4873078_characterizeImage_898326_2.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898326/7d3152b1-6692-4435-9366-c053e4873078_characterizeImage_898326_2.stderr parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 75 submitted for App calibrate, waiting on task 74 parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 75 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 76 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash 3afd370b25175e2b9b252e3249fab9ea parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394ba80e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 67 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898312/8c9728f4-fe0e-467c-9118-4c14df162c3d_characterizeImage_898312_2.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898312/8c9728f4-fe0e-467c-9118-4c14df162c3d_characterizeImage_898312_2.stderr parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 77 submitted for App calibrate, waiting on task 76 parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 77 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 78 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 78 has memoization hash 9352761a2d2f84142e4de4fafc6a6e0e parsl.dataflow.memoization INFO: Task 78 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394ba8180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 68 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898349/38416ffd-f62d-4e69-b703-f02832de1aea_characterizeImage_898349_2.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898349/38416ffd-f62d-4e69-b703-f02832de1aea_characterizeImage_898349_2.stderr parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 79 submitted for App calibrate, waiting on task 78 parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 79 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 80 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 80 has memoization hash 1493f5d7eb240fc798286fc52113f9e2 parsl.dataflow.memoization INFO: Task 80 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394ba82c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 69 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898317/f42869ae-e0a4-4695-a94d-eae09f51d55c_characterizeImage_898317_2.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898317/f42869ae-e0a4-4695-a94d-eae09f51d55c_characterizeImage_898317_2.stderr parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 81 submitted for App calibrate, waiting on task 80 parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 81 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 82 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash 654acdcd0a0eda691b5063d512516b42 parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394ba84a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 70 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898358/e8ef6efb-7ecf-4b6d-a66f-16586850bca6_characterizeImage_898358_2.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898358/e8ef6efb-7ecf-4b6d-a66f-16586850bca6_characterizeImage_898358_2.stderr parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 83 submitted for App calibrate, waiting on task 82 parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 83 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 84 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 84 has memoization hash 6c1c24bc55d30171015ad7bedde0f64d parsl.dataflow.memoization INFO: Task 84 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394ba8680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 71 parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898323/5133c2f4-b9fe-4bf7-9ff3-9a5d3df60280_characterizeImage_898323_2.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898323/5133c2f4-b9fe-4bf7-9ff3-9a5d3df60280_characterizeImage_898323_2.stderr parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 85 submitted for App calibrate, waiting on task 84 parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 85 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 86 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 86 has memoization hash 28c150fec6c19bf55aa1b4c8944de0f6 parsl.dataflow.memoization INFO: Task 86 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394ba8a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 72 parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898352/ea28b3c9-e970-4b1f-9393-bd5bf5dd9520_characterizeImage_898352_2.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898352/ea28b3c9-e970-4b1f-9393-bd5bf5dd9520_characterizeImage_898352_2.stderr parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 87 submitted for App calibrate, waiting on task 86 parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 87 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 88 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 88 has memoization hash 628def22a10cce152316023db3117880 parsl.dataflow.memoization INFO: Task 88 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394ba8860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 73 parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898302/0b03baeb-44ab-4f8f-b76b-8fe5966ddbd8_characterizeImage_898302_2.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898302/0b03baeb-44ab-4f8f-b76b-8fe5966ddbd8_characterizeImage_898302_2.stderr parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 89 submitted for App calibrate, waiting on task 88 parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 89 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 90 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 90 has memoization hash 21c96c8f13668542a720b71501d6e6d6 parsl.dataflow.memoization INFO: Task 90 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394ba8c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 74 parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898321/b6b263c7-e416-493b-a529-87f8e2894e51_characterizeImage_898321_2.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898321/b6b263c7-e416-493b-a529-87f8e2894e51_characterizeImage_898321_2.stderr parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 91 submitted for App calibrate, waiting on task 90 parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 91 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 92 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash 261ae0ee6490810520ccbcac71c2312b parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394ba8e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 75 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898297/669fdf04-b21b-4b07-b670-2caae0e92c0d_characterizeImage_898297_2.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898297/669fdf04-b21b-4b07-b670-2caae0e92c0d_characterizeImage_898297_2.stderr parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 93 submitted for App calibrate, waiting on task 92 parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 93 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 94 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 94 has memoization hash f59722a9d505c1b91a9d5f565af0f069 parsl.dataflow.memoization INFO: Task 94 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394ba8fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 76 parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898332/b22b4ad9-5899-4a01-8fed-8564cd53ba37_characterizeImage_898332_2.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898332/b22b4ad9-5899-4a01-8fed-8564cd53ba37_characterizeImage_898332_2.stderr parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 95 submitted for App calibrate, waiting on task 94 parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 95 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 96 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 96 has memoization hash 4cd75d7ac7ccaa697b997bf5d33d8730 parsl.dataflow.memoization INFO: Task 96 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394ba93a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 77 parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898328/d604c1ba-966b-4f42-8457-f96d32e53430_characterizeImage_898328_2.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898328/d604c1ba-966b-4f42-8457-f96d32e53430_characterizeImage_898328_2.stderr parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 97 submitted for App calibrate, waiting on task 96 parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 97 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 98 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 98 has memoization hash 4e8b658cbaab8e531dc6f1e523d932e2 parsl.dataflow.memoization INFO: Task 98 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394ba91c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 78 parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898346/8b7896f6-d177-4d6d-977b-67c894156eee_characterizeImage_898346_2.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898346/8b7896f6-d177-4d6d-977b-67c894156eee_characterizeImage_898346_2.stderr parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 99 submitted for App calibrate, waiting on task 98 parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 99 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 100 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash fd409323414c1a1d0f6cd6241a0ac7a1 parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394ba9580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 79 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898373/38ee9896-518b-48b2-a68f-de2b4b9536df_characterizeImage_898373_2.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898373/38ee9896-518b-48b2-a68f-de2b4b9536df_characterizeImage_898373_2.stderr parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 101 submitted for App calibrate, waiting on task 100 parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 101 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 102 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 102 has memoization hash 64e7fc366a0525f9369946feb5abbf09 parsl.dataflow.memoization INFO: Task 102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394ba9760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 80 parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898365/041b194e-fef4-41c1-9660-e78ee710ec76_characterizeImage_898365_2.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898365/041b194e-fef4-41c1-9660-e78ee710ec76_characterizeImage_898365_2.stderr parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 103 submitted for App calibrate, waiting on task 102 parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 103 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 104 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 104 has memoization hash f00b1dd700441f6be9a0dd3f6d51e9a3 parsl.dataflow.memoization INFO: Task 104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394ba9940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 81 parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898370/56990dd5-5747-4b9a-92a7-2fefb7fcd19b_characterizeImage_898370_2.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898370/56990dd5-5747-4b9a-92a7-2fefb7fcd19b_characterizeImage_898370_2.stderr parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 105 submitted for App calibrate, waiting on task 104 parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 105 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 106 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash 10e68e6a2238ca602f91fc1a6083824b parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394ba9d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 82 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898356/1dde0e09-6994-4851-bffb-41645b1063f6_characterizeImage_898356_2.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898356/1dde0e09-6994-4851-bffb-41645b1063f6_characterizeImage_898356_2.stderr parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 107 submitted for App calibrate, waiting on task 106 parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 107 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 108 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 108 has memoization hash 004a77f79aec5d3f5723722d59e73a05 parsl.dataflow.memoization INFO: Task 108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394ba9b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 83 parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898330/60a380f1-e740-4766-baed-0adb4a25953e_characterizeImage_898330_2.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898330/60a380f1-e740-4766-baed-0adb4a25953e_characterizeImage_898330_2.stderr parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 109 submitted for App calibrate, waiting on task 108 parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 109 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 110 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 110 has memoization hash bc1139a2902a49b0759292637d4b2394 parsl.dataflow.memoization INFO: Task 110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394ba9ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 84 parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898324/b42216a1-ed0d-4817-b690-a87e5f1b15a6_characterizeImage_898324_2.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898324/b42216a1-ed0d-4817-b690-a87e5f1b15a6_characterizeImage_898324_2.stderr parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 111 submitted for App calibrate, waiting on task 110 parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 111 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 112 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 112 has memoization hash 681c26047d26280295a0ebbee5ec6622 parsl.dataflow.memoization INFO: Task 112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394baa0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 85 parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898322/a639d512-32dc-4105-9ea8-df8022357a3b_characterizeImage_898322_2.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898322/a639d512-32dc-4105-9ea8-df8022357a3b_characterizeImage_898322_2.stderr parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 113 submitted for App calibrate, waiting on task 112 parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 113 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 114 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 114 has memoization hash d714e2248c9ad715ae07492a0747914d parsl.dataflow.memoization INFO: Task 114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394baa2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 86 parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898303/a533e21c-7e5e-45ca-aee7-0090ab76421b_characterizeImage_898303_2.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898303/a533e21c-7e5e-45ca-aee7-0090ab76421b_characterizeImage_898303_2.stderr parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 115 submitted for App calibrate, waiting on task 114 parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 115 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 116 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 116 has memoization hash 66d86101b452de8bf87175fc34ff50a4 parsl.dataflow.memoization INFO: Task 116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394baa660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 87 parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898371/6677da51-c7e3-4e99-a0d2-010a7e7f57b9_characterizeImage_898371_2.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898371/6677da51-c7e3-4e99-a0d2-010a7e7f57b9_characterizeImage_898371_2.stderr parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 117 submitted for App calibrate, waiting on task 116 parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 117 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 118 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash 649fc220bc2bffba313897db8374eb97 parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394baa480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 88 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898345/cbf6c8ea-e38c-4321-85ca-0eac41820570_characterizeImage_898345_2.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898345/cbf6c8ea-e38c-4321-85ca-0eac41820570_characterizeImage_898345_2.stderr parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 119 submitted for App calibrate, waiting on task 118 parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 119 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 120 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 120 has memoization hash 4f1959275519093a86082fc455f48e67 parsl.dataflow.memoization INFO: Task 120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394baa840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 89 parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898369/2fe593b4-90ae-4c32-b322-54407729f112_characterizeImage_898369_2.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898369/2fe593b4-90ae-4c32-b322-54407729f112_characterizeImage_898369_2.stderr parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 121 submitted for App calibrate, waiting on task 120 parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 121 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 122 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 122 has memoization hash a5f8477f44414355d2876740a66b990a parsl.dataflow.memoization INFO: Task 122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394baaa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 90 parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898310/f0d0662d-41f6-4456-a487-ef4267af8fca_characterizeImage_898310_2.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898310/f0d0662d-41f6-4456-a487-ef4267af8fca_characterizeImage_898310_2.stderr parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 123 submitted for App calibrate, waiting on task 122 parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 123 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 124 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 124 has memoization hash d2ee61bdc2b93667ff095c09f615fac2 parsl.dataflow.memoization INFO: Task 124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394baac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 91 parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898327/78d3fffa-c228-475d-941c-2f6d356c99b2_characterizeImage_898327_2.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898327/78d3fffa-c228-475d-941c-2f6d356c99b2_characterizeImage_898327_2.stderr parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 125 submitted for App calibrate, waiting on task 124 parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 125 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 126 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 126 has memoization hash b38f6a39639aa9b0ee832c766689c8fa parsl.dataflow.memoization INFO: Task 126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394baafc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 92 parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898295/dfde6b77-0194-4de1-b802-86a134975378_characterizeImage_898295_2.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898295/dfde6b77-0194-4de1-b802-86a134975378_characterizeImage_898295_2.stderr parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 127 submitted for App calibrate, waiting on task 126 parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 127 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 128 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 128 has memoization hash e714a8c3f307643a6b4d1d75d6a45e73 parsl.dataflow.memoization INFO: Task 128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394baade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 93 parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898376/b14bc756-1590-4a0e-b075-74eecaae062f_characterizeImage_898376_2.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898376/b14bc756-1590-4a0e-b075-74eecaae062f_characterizeImage_898376_2.stderr parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 129 submitted for App calibrate, waiting on task 128 parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 129 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 130 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 130 has memoization hash e2d9b4af3199bf1a4af8b80b93803cbb parsl.dataflow.memoization INFO: Task 130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bab1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 94 parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898342/e6cece1d-a943-4e40-ae51-8391141a889a_characterizeImage_898342_2.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898342/e6cece1d-a943-4e40-ae51-8391141a889a_characterizeImage_898342_2.stderr parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 131 submitted for App calibrate, waiting on task 130 parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 131 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 132 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 132 has memoization hash 46027d2f5a228a5f097bec7cb9256df5 parsl.dataflow.memoization INFO: Task 132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bab380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 95 parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898333/14edeee9-1368-4638-9bd8-cad4be9f250f_characterizeImage_898333_2.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898333/14edeee9-1368-4638-9bd8-cad4be9f250f_characterizeImage_898333_2.stderr parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 133 submitted for App calibrate, waiting on task 132 parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 133 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 134 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 134 has memoization hash eef3adc3e008b9c140003faf6a90c2b7 parsl.dataflow.memoization INFO: Task 134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bab560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 96 parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898293/3d3968e4-c012-4124-a509-f38e1e0d9fa8_characterizeImage_898293_2.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898293/3d3968e4-c012-4124-a509-f38e1e0d9fa8_characterizeImage_898293_2.stderr parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 135 submitted for App calibrate, waiting on task 134 parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 135 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 136 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 136 has memoization hash a4b8335b5d19157dd964fbb1e42fd80a parsl.dataflow.memoization INFO: Task 136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bab920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 97 parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898319/97c032f7-fed0-48f5-98c5-8ef97841e90c_characterizeImage_898319_2.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898319/97c032f7-fed0-48f5-98c5-8ef97841e90c_characterizeImage_898319_2.stderr parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 137 submitted for App calibrate, waiting on task 136 parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 137 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 138 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 138 has memoization hash ff5b909ecfc00a8ede54916c785b683f parsl.dataflow.memoization INFO: Task 138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bab740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 98 parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898339/91647e66-469e-416e-a268-f8eed54c914e_characterizeImage_898339_2.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898339/91647e66-469e-416e-a268-f8eed54c914e_characterizeImage_898339_2.stderr parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 139 submitted for App calibrate, waiting on task 138 parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 139 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 140 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 140 has memoization hash a537e402d903b036ed90a11e2e1b344e parsl.dataflow.memoization INFO: Task 140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394babb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 99 parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898318/fc5dc045-bf0e-43d6-95e3-1b7b0a0635f7_characterizeImage_898318_2.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898318/fc5dc045-bf0e-43d6-95e3-1b7b0a0635f7_characterizeImage_898318_2.stderr parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 141 submitted for App calibrate, waiting on task 140 parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 141 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 142 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 142 has memoization hash c3c2e67415f37b774d20ed2a974e664b parsl.dataflow.memoization INFO: Task 142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394babce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 100 parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898287/3fded289-8eed-4ce8-b353-f1e8205743a1_characterizeImage_898287_2.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898287/3fded289-8eed-4ce8-b353-f1e8205743a1_characterizeImage_898287_2.stderr parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 143 submitted for App calibrate, waiting on task 142 parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 143 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 144 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 144 has memoization hash 9c97e3fa9e28b23d513b63c136164642 parsl.dataflow.memoization INFO: Task 144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bd8040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 101 parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898337/3bedf328-bece-48f2-a146-6fdd66b27359_characterizeImage_898337_2.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898337/3bedf328-bece-48f2-a146-6fdd66b27359_characterizeImage_898337_2.stderr parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 145 submitted for App calibrate, waiting on task 144 parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 145 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 146 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 146 has memoization hash 2b0511c6c1b4436dd88b008f34f8d4a2 parsl.dataflow.memoization INFO: Task 146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bd82c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 102 parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898362/273324db-4d70-46d8-a1a7-f4b198e1783f_characterizeImage_898362_2.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898362/273324db-4d70-46d8-a1a7-f4b198e1783f_characterizeImage_898362_2.stderr parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 147 submitted for App calibrate, waiting on task 146 parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 147 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 148 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 148 has memoization hash 76e251355064399dc11343e930b21e23 parsl.dataflow.memoization INFO: Task 148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bd80e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 103 parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898360/6f8c9e81-9a6b-4554-8d15-ce766f7c0e51_characterizeImage_898360_2.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898360/6f8c9e81-9a6b-4554-8d15-ce766f7c0e51_characterizeImage_898360_2.stderr parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 149 submitted for App calibrate, waiting on task 148 parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 149 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 150 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 150 has memoization hash e0f7a4f2a8ae6516d8b754d10ea1c74d parsl.dataflow.memoization INFO: Task 150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bd84a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 104 parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898344/ad25b03a-d309-4099-a4c8-f0db38b05219_characterizeImage_898344_2.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898344/ad25b03a-d309-4099-a4c8-f0db38b05219_characterizeImage_898344_2.stderr parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 151 submitted for App calibrate, waiting on task 150 parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 151 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 152 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 152 has memoization hash 407875da43491a938cd49d59b18c6273 parsl.dataflow.memoization INFO: Task 152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bd8680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 105 parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898354/0704352b-dd37-4eb0-b3ca-3928ab62298f_characterizeImage_898354_2.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898354/0704352b-dd37-4eb0-b3ca-3928ab62298f_characterizeImage_898354_2.stderr parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 153 submitted for App calibrate, waiting on task 152 parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 153 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 154 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 154 has memoization hash eb57123a91a4dde1df4fd39a8b6f90d8 parsl.dataflow.memoization INFO: Task 154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bd8860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 106 parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898294/f5b9f324-1334-4e2e-a191-f3b12aaa5965_characterizeImage_898294_2.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898294/f5b9f324-1334-4e2e-a191-f3b12aaa5965_characterizeImage_898294_2.stderr parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 155 submitted for App calibrate, waiting on task 154 parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 155 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 156 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 156 has memoization hash b67bc97637cec3612a5869fcc9cf54f0 parsl.dataflow.memoization INFO: Task 156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bd8c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 107 parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898286/298e4d3d-7adc-4228-b469-702d4902d5d1_characterizeImage_898286_2.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898286/298e4d3d-7adc-4228-b469-702d4902d5d1_characterizeImage_898286_2.stderr parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 157 submitted for App calibrate, waiting on task 156 parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 157 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 158 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 158 has memoization hash a24c7ea12e2c2d17472c2e285c904e3c parsl.dataflow.memoization INFO: Task 158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bd8a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 108 parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898368/402adb10-54c3-4a78-9606-4fcca2726950_characterizeImage_898368_2.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898368/402adb10-54c3-4a78-9606-4fcca2726950_characterizeImage_898368_2.stderr parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 159 submitted for App calibrate, waiting on task 158 parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 159 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 160 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 160 has memoization hash 45bc3cff8c8d52a100a544bcbdac6f1e parsl.dataflow.memoization INFO: Task 160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bd8e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 109 parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898307/600d84f7-6e60-4406-8464-e4a9809afada_characterizeImage_898307_2.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898307/600d84f7-6e60-4406-8464-e4a9809afada_characterizeImage_898307_2.stderr parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 161 submitted for App calibrate, waiting on task 160 parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 161 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 162 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 162 has memoization hash ea6a56b7ace35c002723f8d6242d2385 parsl.dataflow.memoization INFO: Task 162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bd8fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 110 parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898315/e2b2c7ea-f52d-42e1-a0fd-3e90ac60d7e2_characterizeImage_898315_2.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898315/e2b2c7ea-f52d-42e1-a0fd-3e90ac60d7e2_characterizeImage_898315_2.stderr parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 163 submitted for App calibrate, waiting on task 162 parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 163 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 164 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 164 has memoization hash 308fe0cd5112c0272d33c1f37c2dfe89 parsl.dataflow.memoization INFO: Task 164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bd91c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 111 parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898350/6e8de21e-cc46-48e9-babd-fad71dbccbc4_characterizeImage_898350_2.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898350/6e8de21e-cc46-48e9-babd-fad71dbccbc4_characterizeImage_898350_2.stderr parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 165 submitted for App calibrate, waiting on task 164 parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 165 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 166 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 166 has memoization hash 9e37b0df012723d5c38d6fd39629e374 parsl.dataflow.memoization INFO: Task 166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bd9580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 112 parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898300/b94fef20-f629-47af-93e6-c9f2a3a9d58e_characterizeImage_898300_2.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898300/b94fef20-f629-47af-93e6-c9f2a3a9d58e_characterizeImage_898300_2.stderr parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 167 submitted for App calibrate, waiting on task 166 parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 167 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 168 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 168 has memoization hash b04f9d26fa7a7033635a9284cede03de parsl.dataflow.memoization INFO: Task 168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bd93a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 113 parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898353/8b3003ab-d982-40fb-9a29-d42d3753c93a_characterizeImage_898353_2.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898353/8b3003ab-d982-40fb-9a29-d42d3753c93a_characterizeImage_898353_2.stderr parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 169 submitted for App calibrate, waiting on task 168 parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 169 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 170 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 170 has memoization hash bb3d28e661ccb7a9e2497ea69d198be1 parsl.dataflow.memoization INFO: Task 170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bd9760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 170 try 0 launched on executor multi with executor id 114 parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898381/def23eee-f468-417e-a13d-2c3590830f1a_characterizeImage_898381_2.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898381/def23eee-f468-417e-a13d-2c3590830f1a_characterizeImage_898381_2.stderr parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 171 submitted for App calibrate, waiting on task 170 parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 171 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 172 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 172 has memoization hash 82a7632dea989fb59b9debf483e8f257 parsl.dataflow.memoization INFO: Task 172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bd9940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 115 parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898291/3f08e496-3c1c-4e7b-81b8-e00bfd49915c_characterizeImage_898291_2.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898291/3f08e496-3c1c-4e7b-81b8-e00bfd49915c_characterizeImage_898291_2.stderr parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 173 submitted for App calibrate, waiting on task 172 parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 173 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 174 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 174 has memoization hash 53d95f458579079a7083b5d9467898cf parsl.dataflow.memoization INFO: Task 174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bd9b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 116 parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898325/a1d83ff3-a248-4b6d-a09b-46c0ec2b1db9_characterizeImage_898325_2.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898325/a1d83ff3-a248-4b6d-a09b-46c0ec2b1db9_characterizeImage_898325_2.stderr parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 175 submitted for App calibrate, waiting on task 174 parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 175 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 176 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 176 has memoization hash 789a3cf4fbdb6926020ca79be33c4a3b parsl.dataflow.memoization INFO: Task 176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bd9ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 117 parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898289/826cbbf6-3f98-46d6-8550-55ac105a4087_characterizeImage_898289_2.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898289/826cbbf6-3f98-46d6-8550-55ac105a4087_characterizeImage_898289_2.stderr parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 177 submitted for App calibrate, waiting on task 176 parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 177 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 178 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 178 has memoization hash 5e6ed418d182a60ebe3b6707af1f3fb7 parsl.dataflow.memoization INFO: Task 178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bd9d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 118 parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898316/859c2f84-b5d9-4766-8e01-03ce74cd180a_characterizeImage_898316_2.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898316/859c2f84-b5d9-4766-8e01-03ce74cd180a_characterizeImage_898316_2.stderr parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 179 submitted for App calibrate, waiting on task 178 parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 179 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 180 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 180 has memoization hash 83183bddadbe819dd868ad0180af8efb parsl.dataflow.memoization INFO: Task 180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bda0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 119 parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898340/d4cf30dc-0e16-4a99-b41a-89a591835063_characterizeImage_898340_2.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898340/d4cf30dc-0e16-4a99-b41a-89a591835063_characterizeImage_898340_2.stderr parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 181 submitted for App calibrate, waiting on task 180 parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 181 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 182 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 182 has memoization hash 187009d2c954513b50496652950b3e61 parsl.dataflow.memoization INFO: Task 182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bda2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 120 parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898320/0d271a51-6393-4e3c-8de9-ccd83369b17f_characterizeImage_898320_2.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898320/0d271a51-6393-4e3c-8de9-ccd83369b17f_characterizeImage_898320_2.stderr parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 183 submitted for App calibrate, waiting on task 182 parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 183 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 184 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 184 has memoization hash c3b73f90ebf119d57db71ad1aae6ce4e parsl.dataflow.memoization INFO: Task 184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bda480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 121 parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898374/7d868f31-aea1-4df3-bca4-b4f657a601a8_characterizeImage_898374_2.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898374/7d868f31-aea1-4df3-bca4-b4f657a601a8_characterizeImage_898374_2.stderr parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 185 submitted for App calibrate, waiting on task 184 parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 185 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 186 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 186 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 186 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 186 has memoization hash e6bf795532176e2d27dad486567f1e64 parsl.dataflow.memoization INFO: Task 186 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bda840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 186 try 0 launched on executor multi with executor id 122 parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898331/c0c18cff-9b6d-4db0-9644-84341b2c3bbd_characterizeImage_898331_2.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898331/c0c18cff-9b6d-4db0-9644-84341b2c3bbd_characterizeImage_898331_2.stderr parsl.dataflow.dflow DEBUG: Task 187 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 187 submitted for App calibrate, waiting on task 186 parsl.dataflow.dflow DEBUG: Task 187 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 187 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 188 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 188 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 188 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 188 has memoization hash 3732fc702b75d329c601489899e3f640 parsl.dataflow.memoization INFO: Task 188 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bda660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 188 try 0 launched on executor multi with executor id 123 parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898304/f4fe1812-be36-41cd-8197-d661a0fcd8c9_characterizeImage_898304_2.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898304/f4fe1812-be36-41cd-8197-d661a0fcd8c9_characterizeImage_898304_2.stderr parsl.dataflow.dflow DEBUG: Task 189 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 189 submitted for App calibrate, waiting on task 188 parsl.dataflow.dflow DEBUG: Task 189 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 189 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 190 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 190 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 190 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 190 has memoization hash 3492ba1a18c1958aa8dd61434111c302 parsl.dataflow.memoization INFO: Task 190 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bdaa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 190 try 0 launched on executor multi with executor id 124 parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898341/1ff0f251-6bb7-4969-aad8-00b357877c1c_characterizeImage_898341_2.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898341/1ff0f251-6bb7-4969-aad8-00b357877c1c_characterizeImage_898341_2.stderr parsl.dataflow.dflow DEBUG: Task 191 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 191 submitted for App calibrate, waiting on task 190 parsl.dataflow.dflow DEBUG: Task 191 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 191 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 192 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 192 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 192 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 192 has memoization hash 20726f9423ee27a4b04345ffe73518b7 parsl.dataflow.memoization INFO: Task 192 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bdac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 192 try 0 launched on executor multi with executor id 125 parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898338/cb25c482-36e9-4bee-a408-37e1611db7cf_characterizeImage_898338_2.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898338/cb25c482-36e9-4bee-a408-37e1611db7cf_characterizeImage_898338_2.stderr parsl.dataflow.dflow DEBUG: Task 193 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 193 submitted for App calibrate, waiting on task 192 parsl.dataflow.dflow DEBUG: Task 193 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 193 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 194 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 194 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 194 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 194 has memoization hash c862e51217a6299650d85a3e39d0d8d7 parsl.dataflow.memoization INFO: Task 194 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bdade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 194 try 0 launched on executor multi with executor id 126 parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898366/883e1fdb-0061-4803-a77b-f6a737d0eb4d_characterizeImage_898366_2.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898366/883e1fdb-0061-4803-a77b-f6a737d0eb4d_characterizeImage_898366_2.stderr parsl.dataflow.dflow DEBUG: Task 195 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 195 submitted for App calibrate, waiting on task 194 parsl.dataflow.dflow DEBUG: Task 195 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 195 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 196 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 196 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 196 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 196 has memoization hash 7c164b614258f56e5d1018602d2c6b22 parsl.dataflow.memoization INFO: Task 196 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bdb1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 196 try 0 launched on executor multi with executor id 127 parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898313/8efe628c-fbcb-4620-b8a1-da84903ac46a_characterizeImage_898313_2.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898313/8efe628c-fbcb-4620-b8a1-da84903ac46a_characterizeImage_898313_2.stderr parsl.dataflow.dflow DEBUG: Task 197 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 197 submitted for App calibrate, waiting on task 196 parsl.dataflow.dflow DEBUG: Task 197 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 197 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 198 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 198 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 198 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 198 has memoization hash 9860e33d4639157e87123b78bcea19b9 parsl.dataflow.memoization INFO: Task 198 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bdafc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 198 try 0 launched on executor multi with executor id 128 parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898382/fdd9caee-a9d8-47b5-8d09-8b6e064291a9_characterizeImage_898382_2.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898382/fdd9caee-a9d8-47b5-8d09-8b6e064291a9_characterizeImage_898382_2.stderr parsl.dataflow.dflow DEBUG: Task 199 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 199 submitted for App calibrate, waiting on task 198 parsl.dataflow.dflow DEBUG: Task 199 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 199 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 200 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 200 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 200 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 200 has memoization hash 98a6b5e9eb625aa688a4be5b26f90646 parsl.dataflow.memoization INFO: Task 200 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bdb380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 200 try 0 launched on executor multi with executor id 129 parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898347/cc7819d0-5467-422c-b477-88c64cf6b601_characterizeImage_898347_2.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898347/cc7819d0-5467-422c-b477-88c64cf6b601_characterizeImage_898347_2.stderr parsl.dataflow.dflow DEBUG: Task 201 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 201 submitted for App calibrate, waiting on task 200 parsl.dataflow.dflow DEBUG: Task 201 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 201 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 202 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 202 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 202 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 202 has memoization hash 5856865b095f60519def504b9848d2e8 parsl.dataflow.memoization INFO: Task 202 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bdb560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 202 try 0 launched on executor multi with executor id 130 parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898305/6eb2b7de-5132-4dab-852e-cf7330151622_characterizeImage_898305_2.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898305/6eb2b7de-5132-4dab-852e-cf7330151622_characterizeImage_898305_2.stderr parsl.dataflow.dflow DEBUG: Task 203 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 203 submitted for App calibrate, waiting on task 202 parsl.dataflow.dflow DEBUG: Task 203 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 203 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 204 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 204 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 204 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 204 has memoization hash 24d2c9716569392e6d623f504981bd3b parsl.dataflow.memoization INFO: Task 204 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bdb740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 204 try 0 launched on executor multi with executor id 131 parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898308/243ba183-700c-4b37-9a39-4b8717e68bde_characterizeImage_898308_2.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898308/243ba183-700c-4b37-9a39-4b8717e68bde_characterizeImage_898308_2.stderr parsl.dataflow.dflow DEBUG: Task 205 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 205 submitted for App calibrate, waiting on task 204 parsl.dataflow.dflow DEBUG: Task 205 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 205 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 206 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 206 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 206 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 206 has memoization hash 47cbca9dbf88bdcdc02765303207aee9 parsl.dataflow.memoization INFO: Task 206 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bdbb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 206 try 0 launched on executor multi with executor id 132 parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898292/3ca39bab-f6c2-4bd5-9b58-06747c252ce8_characterizeImage_898292_2.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898292/3ca39bab-f6c2-4bd5-9b58-06747c252ce8_characterizeImage_898292_2.stderr parsl.dataflow.dflow DEBUG: Task 207 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 207 submitted for App calibrate, waiting on task 206 parsl.dataflow.dflow DEBUG: Task 207 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 207 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 208 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 208 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 208 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 208 has memoization hash 87aac7584d5218dd9b19f278ccc6e0d5 parsl.dataflow.memoization INFO: Task 208 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bdb920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 208 try 0 launched on executor multi with executor id 133 parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898378/0995fe10-ea29-4d67-a721-fb8b6cd22e66_characterizeImage_898378_2.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898378/0995fe10-ea29-4d67-a721-fb8b6cd22e66_characterizeImage_898378_2.stderr parsl.dataflow.dflow DEBUG: Task 209 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 209 submitted for App calibrate, waiting on task 208 parsl.dataflow.dflow DEBUG: Task 209 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 209 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 210 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 210 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 210 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 210 has memoization hash 37c00a91549aa1477fe31946f97a7c5f parsl.dataflow.memoization INFO: Task 210 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bdbce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 210 try 0 launched on executor multi with executor id 134 parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898379/64202fb7-6336-4e9f-8200-02d1651dce8f_characterizeImage_898379_2.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898379/64202fb7-6336-4e9f-8200-02d1651dce8f_characterizeImage_898379_2.stderr parsl.dataflow.dflow DEBUG: Task 211 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 211 submitted for App calibrate, waiting on task 210 parsl.dataflow.dflow DEBUG: Task 211 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 211 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 212 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 212 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 212 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 212 has memoization hash 5dcdaaa9962234e2d4da66b378702105 parsl.dataflow.memoization INFO: Task 212 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394c0c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 212 try 0 launched on executor multi with executor id 135 parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898336/59045ea5-e278-4d7d-a033-56463a2f5fac_characterizeImage_898336_2.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898336/59045ea5-e278-4d7d-a033-56463a2f5fac_characterizeImage_898336_2.stderr parsl.dataflow.dflow DEBUG: Task 213 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 213 submitted for App calibrate, waiting on task 212 parsl.dataflow.dflow DEBUG: Task 213 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 213 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 214 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 214 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 214 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 214 has memoization hash 9e891114e331dc0777c030114d61ec77 parsl.dataflow.memoization INFO: Task 214 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394c0c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 214 try 0 launched on executor multi with executor id 136 parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898351/8862f32c-0af5-4ab1-84b5-911135b20c05_characterizeImage_898351_2.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898351/8862f32c-0af5-4ab1-84b5-911135b20c05_characterizeImage_898351_2.stderr parsl.dataflow.dflow DEBUG: Task 215 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 215 submitted for App calibrate, waiting on task 214 parsl.dataflow.dflow DEBUG: Task 215 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 215 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 216 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 216 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 216 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 216 has memoization hash f714afe43fbdf6119189762e695b5bfe parsl.dataflow.memoization INFO: Task 216 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394c0c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 216 try 0 launched on executor multi with executor id 137 parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898363/15ceb904-a4e7-4b48-ac94-ffecf8eeed6b_characterizeImage_898363_2.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898363/15ceb904-a4e7-4b48-ac94-ffecf8eeed6b_characterizeImage_898363_2.stderr parsl.dataflow.dflow DEBUG: Task 217 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 217 submitted for App calibrate, waiting on task 216 parsl.dataflow.dflow DEBUG: Task 217 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 217 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 218 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 218 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 218 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 218 has memoization hash 7cf33828fc86323e1122578488096e86 parsl.dataflow.memoization INFO: Task 218 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394c0c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 218 try 0 launched on executor multi with executor id 138 parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898372/92791b56-4fa4-4dbb-928e-3fbe354d21b6_characterizeImage_898372_2.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898372/92791b56-4fa4-4dbb-928e-3fbe354d21b6_characterizeImage_898372_2.stderr parsl.dataflow.dflow DEBUG: Task 219 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 219 submitted for App calibrate, waiting on task 218 parsl.dataflow.dflow DEBUG: Task 219 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 219 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 220 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 220 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 220 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 220 has memoization hash a2683b89b666f946be5df544b264d391 parsl.dataflow.memoization INFO: Task 220 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394c0c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 220 try 0 launched on executor multi with executor id 139 parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898335/281503df-63c1-4918-888e-a90e0e4d90ea_characterizeImage_898335_2.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898335/281503df-63c1-4918-888e-a90e0e4d90ea_characterizeImage_898335_2.stderr parsl.dataflow.dflow DEBUG: Task 221 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 221 submitted for App calibrate, waiting on task 220 parsl.dataflow.dflow DEBUG: Task 221 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 221 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 222 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 222 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 222 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 222 has memoization hash b6a5488867a8c941e81fba6ca799cbd6 parsl.dataflow.memoization INFO: Task 222 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394c0c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 222 try 0 launched on executor multi with executor id 140 parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898299/a64d3126-e727-422d-a274-393d2146b227_characterizeImage_898299_2.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898299/a64d3126-e727-422d-a274-393d2146b227_characterizeImage_898299_2.stderr parsl.dataflow.dflow DEBUG: Task 223 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 223 submitted for App calibrate, waiting on task 222 parsl.dataflow.dflow DEBUG: Task 223 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 223 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 224 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 224 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 224 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 224 has memoization hash 18f9bb934b3848c102c6446fb94941ad parsl.dataflow.memoization INFO: Task 224 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394c0ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 224 try 0 launched on executor multi with executor id 141 parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898359/65636a59-80fb-47c1-8f98-43c4f999b198_characterizeImage_898359_2.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898359/65636a59-80fb-47c1-8f98-43c4f999b198_characterizeImage_898359_2.stderr parsl.dataflow.dflow DEBUG: Task 225 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 225 submitted for App calibrate, waiting on task 224 parsl.dataflow.dflow DEBUG: Task 225 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 225 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 226 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 226 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 226 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 226 has memoization hash 70e29d359f3a79d340340ba3825a7d7c parsl.dataflow.memoization INFO: Task 226 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394c0ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 226 try 0 launched on executor multi with executor id 142 parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898329/9a25d39c-9128-4551-b478-cd5d38257f0d_characterizeImage_898329_2.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898329/9a25d39c-9128-4551-b478-cd5d38257f0d_characterizeImage_898329_2.stderr parsl.dataflow.dflow DEBUG: Task 227 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 227 submitted for App calibrate, waiting on task 226 parsl.dataflow.dflow DEBUG: Task 227 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 227 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 228 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 228 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 228 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 228 has memoization hash 5c901b5f51cce774e36848a802b4353c parsl.dataflow.memoization INFO: Task 228 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394c0cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-13 08:44:51 proc_lsst.multi:146 [INFO] found job 34092 in provider local proc_lsst.multi INFO: found job 34092 in provider local parsl.dataflow.dflow INFO: Parsl task 228 try 0 launched on executor multi with executor id 143 parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898343/af00f831-886a-4c91-aed5-e90dc3c54710_characterizeImage_898343_2.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898343/af00f831-886a-4c91-aed5-e90dc3c54710_characterizeImage_898343_2.stderr parsl.dataflow.dflow DEBUG: Task 229 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 229 submitted for App calibrate, waiting on task 228 parsl.dataflow.dflow DEBUG: Task 229 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 229 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 230 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 230 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 230 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 230 has memoization hash c7b4b8af22ecad1d67932da5957e2f6c parsl.dataflow.memoization INFO: Task 230 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394c0d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 230 try 0 launched on executor multi with executor id 144 parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898367/1d9ca05f-cb75-4481-9d04-7f4a93801fb7_characterizeImage_898367_2.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898367/1d9ca05f-cb75-4481-9d04-7f4a93801fb7_characterizeImage_898367_2.stderr parsl.dataflow.dflow DEBUG: Task 231 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 231 submitted for App calibrate, waiting on task 230 parsl.dataflow.dflow DEBUG: Task 231 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 231 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 232 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 232 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 232 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 232 has memoization hash 88c19e98bc647589cf501c6eef68b0f9 parsl.dataflow.memoization INFO: Task 232 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394c0d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 232 try 0 launched on executor multi with executor id 145 parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898348/8791bf45-0123-47d0-97f9-3f528018f72b_characterizeImage_898348_2.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898348/8791bf45-0123-47d0-97f9-3f528018f72b_characterizeImage_898348_2.stderr parsl.dataflow.dflow DEBUG: Task 233 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 233 submitted for App calibrate, waiting on task 232 parsl.dataflow.dflow DEBUG: Task 233 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 233 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 234 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 234 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 234 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 234 has memoization hash 9847deae22a0da9db67da6a661dd1026 parsl.dataflow.memoization INFO: Task 234 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394c0d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 234 try 0 launched on executor multi with executor id 146 parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898309/ab2a58da-d00a-4358-b4f7-75fcdce6e2e5_characterizeImage_898309_2.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898309/ab2a58da-d00a-4358-b4f7-75fcdce6e2e5_characterizeImage_898309_2.stderr parsl.dataflow.dflow DEBUG: Task 235 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 235 submitted for App calibrate, waiting on task 234 parsl.dataflow.dflow DEBUG: Task 235 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 235 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 236 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 236 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 236 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 236 has memoization hash 6f9c150e9df8027239798bcf46aa33f0 parsl.dataflow.memoization INFO: Task 236 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394c0db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 236 try 0 launched on executor multi with executor id 147 parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898298/a0281b0b-1e39-406c-851a-d3046c548e4c_characterizeImage_898298_2.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898298/a0281b0b-1e39-406c-851a-d3046c548e4c_characterizeImage_898298_2.stderr parsl.dataflow.dflow DEBUG: Task 237 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 237 submitted for App calibrate, waiting on task 236 parsl.dataflow.dflow DEBUG: Task 237 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 237 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 238 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 238 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 238 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 238 has memoization hash 23c3f8d39c7bb146f77fc858d2f88f77 parsl.dataflow.memoization INFO: Task 238 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394c0d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 238 try 0 launched on executor multi with executor id 148 parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898296/a19de25d-9b4d-4608-b993-e76d6f64d771_characterizeImage_898296_2.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898296/a19de25d-9b4d-4608-b993-e76d6f64d771_characterizeImage_898296_2.stderr parsl.dataflow.dflow DEBUG: Task 239 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 239 submitted for App calibrate, waiting on task 238 parsl.dataflow.dflow DEBUG: Task 239 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 239 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 240 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 240 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 240 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 240 has memoization hash 0ce1d3fee239f4d14df437cdacf4f103 parsl.dataflow.memoization INFO: Task 240 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394c0dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 240 try 0 launched on executor multi with executor id 149 parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898361/70cb0c22-8082-4558-a092-2d2526583506_characterizeImage_898361_2.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898361/70cb0c22-8082-4558-a092-2d2526583506_characterizeImage_898361_2.stderr parsl.dataflow.dflow DEBUG: Task 241 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 241 submitted for App calibrate, waiting on task 240 parsl.dataflow.dflow DEBUG: Task 241 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 241 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 242 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 242 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 242 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 242 has memoization hash 4074b8db1a55d97a7306e221d12c8572 parsl.dataflow.memoization INFO: Task 242 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394c0dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 242 try 0 launched on executor multi with executor id 150 parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898314/0c096c3b-1ae9-4535-9515-ed489ad83844_characterizeImage_898314_2.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898314/0c096c3b-1ae9-4535-9515-ed489ad83844_characterizeImage_898314_2.stderr parsl.dataflow.dflow DEBUG: Task 243 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 243 submitted for App calibrate, waiting on task 242 parsl.dataflow.dflow DEBUG: Task 243 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 243 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 244 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 244 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 244 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 244 has memoization hash 157cf90d46774aee3d879c243e268d49 parsl.dataflow.memoization INFO: Task 244 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394c0e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 244 try 0 launched on executor multi with executor id 151 parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898377/73990b84-2916-4ebb-b290-1fbd20e627f9_characterizeImage_898377_2.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898377/73990b84-2916-4ebb-b290-1fbd20e627f9_characterizeImage_898377_2.stderr parsl.dataflow.dflow DEBUG: Task 245 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 245 submitted for App calibrate, waiting on task 244 parsl.dataflow.dflow DEBUG: Task 245 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 245 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 246 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 246 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 246 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 246 has memoization hash 210aca5b3c6c55476d15778b06af4561 parsl.dataflow.memoization INFO: Task 246 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394c0e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 152 parsl.dataflow.strategy DEBUG: Executor multi has 152 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 37 more blocks parsl.executors.status_handling INFO: Scaling out by 37 blocks parsl.executors.status_handling INFO: Allocated block ID 1 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1 2024-03-13 08:44:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 2024-03-13 08:44:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 08:44:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:44:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 08:44:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:44:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-13 08:44:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:44:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 246 try 0 launched on executor multi with executor id 152 parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898357/5a7bd7a2-a9a9-41a8-9bb6-6917a13f6a9b_characterizeImage_898357_2.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898357/5a7bd7a2-a9a9-41a8-9bb6-6917a13f6a9b_characterizeImage_898357_2.stderr parsl.dataflow.dflow DEBUG: Task 247 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 247 submitted for App calibrate, waiting on task 246 parsl.dataflow.dflow DEBUG: Task 247 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 247 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 248 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 248 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 248 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 248 has memoization hash 66068bc7a08514af0d0332311219ddb2 parsl.dataflow.memoization INFO: Task 248 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394c0d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 248 try 0 launched on executor multi with executor id 153 parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898364/54421108-f5fa-4a91-9cb8-668488e2f15d_characterizeImage_898364_2.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898364/54421108-f5fa-4a91-9cb8-668488e2f15d_characterizeImage_898364_2.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 249 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 249 submitted for App calibrate, waiting on task 248 parsl.dataflow.dflow DEBUG: Task 249 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 249 has outstanding dependencies, so launch_if_ready skipping 2024-03-13 08:44:52 proc_lsst.multi:170 [INFO] [multi] job_id 17008896 proc_lsst.multi INFO: [multi] job_id 17008896 2024-03-13 08:44:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:44:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:44:52 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17008896 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17008896 parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 17008896 parsl.executors.status_handling INFO: Allocated block ID 2 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2 2024-03-13 08:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 2024-03-13 08:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 08:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 08:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 08:44:52 proc_lsst.multi:170 [INFO] [multi] job_id 17008898 proc_lsst.multi INFO: [multi] job_id 17008898 2024-03-13 08:44:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-13 08:44:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:44:52 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17008898 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17008898 parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 17008898 parsl.executors.status_handling INFO: Allocated block ID 3 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3 2024-03-13 08:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 2024-03-13 08:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 08:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 08:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-13 08:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 08:44:52 proc_lsst.multi:170 [INFO] [multi] job_id 17008899 proc_lsst.multi INFO: [multi] job_id 17008899 2024-03-13 08:44:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-13 08:44:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:44:52 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17008899 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17008899 parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 17008899 parsl.executors.status_handling INFO: Allocated block ID 4 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4 2024-03-13 08:44:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 2024-03-13 08:44:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 08:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 08:44:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:44:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-13 08:44:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:44:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 08:44:53 proc_lsst.multi:170 [INFO] [multi] job_id 17008900 proc_lsst.multi INFO: [multi] job_id 17008900 2024-03-13 08:44:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-13 08:44:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:44:53 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17008900 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17008900 parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 17008900 parsl.executors.status_handling INFO: Allocated block ID 5 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5 2024-03-13 08:44:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 2024-03-13 08:44:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 08:44:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:44:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 08:44:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:44:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-13 08:44:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:44:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 08:44:53 proc_lsst.multi:170 [INFO] [multi] job_id 17008901 proc_lsst.multi INFO: [multi] job_id 17008901 2024-03-13 08:44:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 08:44:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:44:53 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17008901 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17008901 parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 17008901 parsl.executors.status_handling INFO: Allocated block ID 6 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6 2024-03-13 08:44:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 2024-03-13 08:44:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 08:44:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:44:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 08:44:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:44:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 08:44:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:44:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 08:44:53 proc_lsst.multi:170 [INFO] [multi] job_id 17008902 proc_lsst.multi INFO: [multi] job_id 17008902 2024-03-13 08:44:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-13 08:44:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:44:53 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17008902 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17008902 parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 17008902 parsl.executors.status_handling INFO: Allocated block ID 7 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7 2024-03-13 08:44:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 2024-03-13 08:44:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 08:44:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:44:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 08:44:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:44:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-13 08:44:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:44:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 08:44:54 proc_lsst.multi:170 [INFO] [multi] job_id 17008903 proc_lsst.multi INFO: [multi] job_id 17008903 2024-03-13 08:44:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-13 08:44:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:44:54 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17008903 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17008903 parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 17008903 parsl.executors.status_handling INFO: Allocated block ID 8 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8 2024-03-13 08:44:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 2024-03-13 08:44:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 08:44:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:44:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 08:44:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:44:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-13 08:44:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:44:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 08:44:54 proc_lsst.multi:170 [INFO] [multi] job_id 17008904 proc_lsst.multi INFO: [multi] job_id 17008904 2024-03-13 08:44:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-13 08:44:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:44:54 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17008904 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17008904 parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 17008904 parsl.executors.status_handling INFO: Allocated block ID 9 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9 2024-03-13 08:44:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 2024-03-13 08:44:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 08:44:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:44:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 08:44:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:44:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-13 08:44:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:44:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 08:44:54 proc_lsst.multi:170 [INFO] [multi] job_id 17008905 proc_lsst.multi INFO: [multi] job_id 17008905 2024-03-13 08:44:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-13 08:44:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:44:54 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17008905 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17008905 parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 17008905 parsl.executors.status_handling INFO: Allocated block ID 10 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10 2024-03-13 08:44:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 2024-03-13 08:44:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 08:44:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:44:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 08:44:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:44:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-13 08:44:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:44:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 08:44:55 proc_lsst.multi:170 [INFO] [multi] job_id 17008906 proc_lsst.multi INFO: [multi] job_id 17008906 2024-03-13 08:44:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-13 08:44:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:44:55 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17008906 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17008906 parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 17008906 parsl.executors.status_handling INFO: Allocated block ID 11 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11 2024-03-13 08:44:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 2024-03-13 08:44:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 08:44:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:44:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 08:44:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:44:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-13 08:44:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:44:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 08:44:56 proc_lsst.multi:170 [INFO] [multi] job_id 17008907 proc_lsst.multi INFO: [multi] job_id 17008907 2024-03-13 08:44:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-13 08:44:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:44:56 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17008907 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17008907 parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 17008907 parsl.executors.status_handling INFO: Allocated block ID 12 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12 2024-03-13 08:44:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 2024-03-13 08:44:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 08:44:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:44:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 08:44:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:44:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-13 08:44:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:44:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 08:44:56 proc_lsst.multi:170 [INFO] [multi] job_id 17008908 proc_lsst.multi INFO: [multi] job_id 17008908 2024-03-13 08:44:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-13 08:44:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:44:56 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17008908 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17008908 parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 17008908 parsl.executors.status_handling INFO: Allocated block ID 13 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13 2024-03-13 08:44:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 2024-03-13 08:44:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 08:44:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:44:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 08:44:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:44:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-13 08:44:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:44:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 08:44:57 proc_lsst.multi:170 [INFO] [multi] job_id 17008909 proc_lsst.multi INFO: [multi] job_id 17008909 2024-03-13 08:44:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-13 08:44:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:44:57 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17008909 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17008909 parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 17008909 parsl.executors.status_handling INFO: Allocated block ID 14 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14 2024-03-13 08:44:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 2024-03-13 08:44:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 08:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 08:44:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:44:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-13 08:44:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:44:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 08:44:58 proc_lsst.multi:170 [INFO] [multi] job_id 17008910 proc_lsst.multi INFO: [multi] job_id 17008910 2024-03-13 08:44:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-13 08:44:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:44:58 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17008910 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17008910 parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 17008910 parsl.executors.status_handling INFO: Allocated block ID 15 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15 2024-03-13 08:44:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 2024-03-13 08:44:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 08:44:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:44:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 08:44:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:44:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-13 08:44:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:44:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 08:44:58 proc_lsst.multi:170 [INFO] [multi] job_id 17008911 proc_lsst.multi INFO: [multi] job_id 17008911 2024-03-13 08:44:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-13 08:44:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:44:58 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17008911 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17008911 parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 17008911 parsl.executors.status_handling INFO: Allocated block ID 16 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16 2024-03-13 08:44:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 2024-03-13 08:44:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 08:44:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:44:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 08:44:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:44:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-13 08:44:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:44:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 08:44:59 proc_lsst.multi:170 [INFO] [multi] job_id 17008913 proc_lsst.multi INFO: [multi] job_id 17008913 2024-03-13 08:44:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-13 08:44:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:44:59 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17008913 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17008913 parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 17008913 parsl.executors.status_handling INFO: Allocated block ID 17 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17 2024-03-13 08:44:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 2024-03-13 08:44:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 08:44:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:44:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 08:44:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:44:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-13 08:44:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:44:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 08:44:59 proc_lsst.multi:170 [INFO] [multi] job_id 17008914 proc_lsst.multi INFO: [multi] job_id 17008914 2024-03-13 08:44:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-13 08:44:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:44:59 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17008914 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17008914 parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 17008914 parsl.executors.status_handling INFO: Allocated block ID 18 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18 2024-03-13 08:44:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 2024-03-13 08:44:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 08:44:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:44:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 08:44:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:44:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-13 08:44:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:44:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 08:45:00 proc_lsst.multi:170 [INFO] [multi] job_id 17008915 proc_lsst.multi INFO: [multi] job_id 17008915 2024-03-13 08:45:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-13 08:45:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:45:00 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17008915 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17008915 parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 17008915 parsl.executors.status_handling INFO: Allocated block ID 19 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19 2024-03-13 08:45:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 2024-03-13 08:45:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 08:45:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:45:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 08:45:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:45:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-13 08:45:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:45:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 08:45:01 proc_lsst.multi:170 [INFO] [multi] job_id 17008917 proc_lsst.multi INFO: [multi] job_id 17008917 2024-03-13 08:45:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-13 08:45:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:45:01 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17008917 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17008917 parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 17008917 parsl.executors.status_handling INFO: Allocated block ID 20 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20 2024-03-13 08:45:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 2024-03-13 08:45:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 08:45:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:45:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 08:45:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:45:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-13 08:45:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:45:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 08:45:02 proc_lsst.multi:170 [INFO] [multi] job_id 17008918 proc_lsst.multi INFO: [multi] job_id 17008918 2024-03-13 08:45:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 08:45:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:45:02 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17008918 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17008918 parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 17008918 parsl.executors.status_handling INFO: Allocated block ID 21 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21 2024-03-13 08:45:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 2024-03-13 08:45:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 08:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 08:45:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:45:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 08:45:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:45:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 08:45:03 proc_lsst.multi:170 [INFO] [multi] job_id 17008919 proc_lsst.multi INFO: [multi] job_id 17008919 2024-03-13 08:45:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-13 08:45:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:45:03 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17008919 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17008919 parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 17008919 parsl.executors.status_handling INFO: Allocated block ID 22 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22 2024-03-13 08:45:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 2024-03-13 08:45:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 08:45:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:45:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 08:45:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:45:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-13 08:45:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:45:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 08:45:04 proc_lsst.multi:170 [INFO] [multi] job_id 17008921 proc_lsst.multi INFO: [multi] job_id 17008921 2024-03-13 08:45:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-13 08:45:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:45:04 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17008921 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17008921 parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 17008921 parsl.executors.status_handling INFO: Allocated block ID 23 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23 2024-03-13 08:45:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 2024-03-13 08:45:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 08:45:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:45:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 08:45:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:45:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-13 08:45:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:45:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 08:45:05 proc_lsst.multi:170 [INFO] [multi] job_id 17008922 proc_lsst.multi INFO: [multi] job_id 17008922 2024-03-13 08:45:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-13 08:45:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:45:05 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17008922 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17008922 parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 17008922 parsl.executors.status_handling INFO: Allocated block ID 24 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24 2024-03-13 08:45:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 2024-03-13 08:45:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 08:45:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:45:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 08:45:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:45:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-13 08:45:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:45:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 08:45:06 proc_lsst.multi:170 [INFO] [multi] job_id 17008923 proc_lsst.multi INFO: [multi] job_id 17008923 2024-03-13 08:45:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-13 08:45:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:45:06 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17008923 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17008923 parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 17008923 parsl.executors.status_handling INFO: Allocated block ID 25 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25 2024-03-13 08:45:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 2024-03-13 08:45:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 08:45:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:45:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 08:45:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:45:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-13 08:45:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:45:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 08:45:08 proc_lsst.multi:170 [INFO] [multi] job_id 17008924 proc_lsst.multi INFO: [multi] job_id 17008924 2024-03-13 08:45:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-13 08:45:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:45:08 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17008924 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17008924 parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 17008924 parsl.executors.status_handling INFO: Allocated block ID 26 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26 2024-03-13 08:45:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 2024-03-13 08:45:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 08:45:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:45:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 08:45:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:45:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-13 08:45:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:45:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 08:45:09 proc_lsst.multi:170 [INFO] [multi] job_id 17008925 proc_lsst.multi INFO: [multi] job_id 17008925 2024-03-13 08:45:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-13 08:45:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:45:09 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17008925 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17008925 parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 17008925 parsl.executors.status_handling INFO: Allocated block ID 27 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27 2024-03-13 08:45:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 2024-03-13 08:45:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 08:45:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:45:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 08:45:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:45:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-13 08:45:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:45:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 08:45:10 proc_lsst.multi:170 [INFO] [multi] job_id 17008926 proc_lsst.multi INFO: [multi] job_id 17008926 2024-03-13 08:45:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-13 08:45:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:45:10 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17008926 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17008926 parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 17008926 parsl.executors.status_handling INFO: Allocated block ID 28 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28 2024-03-13 08:45:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 2024-03-13 08:45:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 08:45:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:45:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 08:45:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:45:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-13 08:45:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:45:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 08:45:12 proc_lsst.multi:170 [INFO] [multi] job_id 17008927 proc_lsst.multi INFO: [multi] job_id 17008927 2024-03-13 08:45:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-13 08:45:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:45:12 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17008927 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17008927 parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 17008927 parsl.executors.status_handling INFO: Allocated block ID 29 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29 2024-03-13 08:45:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 2024-03-13 08:45:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 08:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 08:45:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:45:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-13 08:45:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:45:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 08:45:13 proc_lsst.multi:170 [INFO] [multi] job_id 17008930 proc_lsst.multi INFO: [multi] job_id 17008930 2024-03-13 08:45:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-13 08:45:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:45:13 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17008930 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17008930 parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 17008930 parsl.executors.status_handling INFO: Allocated block ID 30 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30 2024-03-13 08:45:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 2024-03-13 08:45:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 08:45:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:45:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 08:45:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:45:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-13 08:45:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:45:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 08:45:15 proc_lsst.multi:170 [INFO] [multi] job_id 17008934 proc_lsst.multi INFO: [multi] job_id 17008934 2024-03-13 08:45:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 08:45:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:45:15 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17008934 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17008934 parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 17008934 parsl.executors.status_handling INFO: Allocated block ID 31 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31 2024-03-13 08:45:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 2024-03-13 08:45:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 08:45:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:45:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 08:45:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:45:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 08:45:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:45:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:45:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-13 08:45:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 08:45:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 08:45:16 proc_lsst.multi:170 [INFO] [multi] job_id 17008943 proc_lsst.multi INFO: [multi] job_id 17008943 2024-03-13 08:45:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:45:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 08:45:16 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17008943 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17008943 parsl.executors.status_handling DEBUG: Launched block 31 on executor multi with job ID 17008943 parsl.executors.status_handling INFO: Allocated block ID 32 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32 2024-03-13 08:45:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 2024-03-13 08:45:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 08:45:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:45:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 08:45:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:45:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 08:45:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:45:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:45:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:45:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 08:45:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 08:45:17 proc_lsst.multi:170 [INFO] [multi] job_id 17008949 proc_lsst.multi INFO: [multi] job_id 17008949 2024-03-13 08:45:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-13 08:45:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 08:45:17 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17008949 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17008949 parsl.executors.status_handling DEBUG: Launched block 32 on executor multi with job ID 17008949 parsl.executors.status_handling INFO: Allocated block ID 33 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33 2024-03-13 08:45:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 2024-03-13 08:45:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 08:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 08:45:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 08:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:45:17 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:45:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-13 08:45:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 08:45:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 08:45:19 proc_lsst.multi:170 [INFO] [multi] job_id 17008950 proc_lsst.multi INFO: [multi] job_id 17008950 2024-03-13 08:45:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-13 08:45:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 08:45:19 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17008950 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17008950 parsl.executors.status_handling DEBUG: Launched block 33 on executor multi with job ID 17008950 parsl.executors.status_handling INFO: Allocated block ID 34 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34 2024-03-13 08:45:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 2024-03-13 08:45:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 08:45:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:45:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 08:45:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:45:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 08:45:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:45:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:45:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-13 08:45:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 08:45:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 08:45:20 proc_lsst.multi:170 [INFO] [multi] job_id 17008951 proc_lsst.multi INFO: [multi] job_id 17008951 2024-03-13 08:45:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-13 08:45:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 08:45:20 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17008951 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17008951 parsl.executors.status_handling DEBUG: Launched block 34 on executor multi with job ID 17008951 parsl.executors.status_handling INFO: Allocated block ID 35 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35 2024-03-13 08:45:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 2024-03-13 08:45:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 08:45:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:45:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 08:45:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:45:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 08:45:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:45:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:45:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-13 08:45:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 08:45:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 08:45:21 proc_lsst.multi:170 [INFO] [multi] job_id 17008952 proc_lsst.multi INFO: [multi] job_id 17008952 2024-03-13 08:45:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 08:45:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 08:45:21 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17008952 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17008952 parsl.executors.status_handling DEBUG: Launched block 35 on executor multi with job ID 17008952 parsl.executors.status_handling INFO: Allocated block ID 36 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36 2024-03-13 08:45:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 2024-03-13 08:45:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 08:45:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:45:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 08:45:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:45:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 08:45:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:45:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:45:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 08:45:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 08:45:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:45:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-13 08:45:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 08:45:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 08:45:23 proc_lsst.multi:170 [INFO] [multi] job_id 17008953 proc_lsst.multi INFO: [multi] job_id 17008953 2024-03-13 08:45:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:45:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 08:45:23 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17008953 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17008953 parsl.executors.status_handling DEBUG: Launched block 36 on executor multi with job ID 17008953 parsl.executors.status_handling INFO: Allocated block ID 37 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37 2024-03-13 08:45:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 2024-03-13 08:45:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 08:45:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:45:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 08:45:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:45:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 08:45:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:45:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:45:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 08:45:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 08:45:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:45:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:45:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 08:45:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 08:45:24 proc_lsst.multi:170 [INFO] [multi] job_id 17008954 proc_lsst.multi INFO: [multi] job_id 17008954 2024-03-13 08:45:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-13 08:45:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 08:45:24 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17008954 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17008954 parsl.executors.status_handling DEBUG: Launched block 37 on executor multi with job ID 17008954 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22761558686992 2024-03-13 08:45:24 proc_lsst.multi:146 [INFO] found job 34092 in provider local proc_lsst.multi INFO: found job 34092 in provider local 2024-03-13 08:45:24 proc_lsst.multi:146 [INFO] found job 17008896 in provider astro proc_lsst.multi INFO: found job 17008896 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 1 try 0 failed parsl.dataflow.dflow INFO: Task 1 marked for retry parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898344/5582ca13-75db-4a41-ab17-540aa0a69df1_calibrate_898344_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898344/5582ca13-75db-4a41-ab17-540aa0a69df1_calibrate_898344_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1 has memoization hash 66ab7b72d091b964779f664b0e32bb80 parsl.dataflow.memoization INFO: Task 1 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b68680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1 try 1 launched on executor multi with executor id 154 parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898344/5582ca13-75db-4a41-ab17-540aa0a69df1_calibrate_898344_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898344/5582ca13-75db-4a41-ab17-540aa0a69df1_calibrate_898344_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:45:26 proc_lsst.multi:146 [INFO] found job 17008898 in provider astro proc_lsst.multi INFO: found job 17008898 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:45:27 proc_lsst.multi:146 [INFO] found job 17008899 in provider astro proc_lsst.multi INFO: found job 17008899 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:45:28 proc_lsst.multi:146 [INFO] found job 17008900 in provider astro proc_lsst.multi INFO: found job 17008900 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 0 try 0 failed parsl.dataflow.dflow INFO: Task 0 marked for retry parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898348/81898b87-3975-4957-9c7a-c96ad661712e_calibrate_898348_31.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898348/81898b87-3975-4957-9c7a-c96ad661712e_calibrate_898348_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 0 has memoization hash 09066a0219e24fdbb84bb3def6e26179 parsl.dataflow.memoization INFO: Task 0 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b682c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 0 try 1 launched on executor multi with executor id 155 parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898348/81898b87-3975-4957-9c7a-c96ad661712e_calibrate_898348_31.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898348/81898b87-3975-4957-9c7a-c96ad661712e_calibrate_898348_31.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 3 try 0 failed parsl.dataflow.dflow INFO: Task 3 marked for retry parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898350/ac90d1dd-77a7-4e35-9414-e727205aec56_calibrate_898350_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898350/ac90d1dd-77a7-4e35-9414-e727205aec56_calibrate_898350_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3 has memoization hash ac45f7da037ff0da36555fa281bb6ff8 parsl.dataflow.memoization INFO: Task 3 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b689a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3 try 1 launched on executor multi with executor id 156 parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898350/ac90d1dd-77a7-4e35-9414-e727205aec56_calibrate_898350_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898350/ac90d1dd-77a7-4e35-9414-e727205aec56_calibrate_898350_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:45:30 proc_lsst.multi:146 [INFO] found job 17008901 in provider astro proc_lsst.multi INFO: found job 17008901 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:45:31 proc_lsst.multi:146 [INFO] found job 17008902 in provider astro proc_lsst.multi INFO: found job 17008902 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:45:32 proc_lsst.multi:146 [INFO] found job 17008903 in provider astro proc_lsst.multi INFO: found job 17008903 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 7 try 0 failed parsl.dataflow.dflow INFO: Task 7 marked for retry parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898342/dc2c0f58-ade0-4f10-9908-3da1526301af_calibrate_898342_31.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898342/dc2c0f58-ade0-4f10-9908-3da1526301af_calibrate_898342_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 7 has memoization hash 7e4a79586e0a0d69281374a58e04e7d9 parsl.dataflow.memoization INFO: Task 7 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b685e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 7 try 1 launched on executor multi with executor id 157 parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898342/dc2c0f58-ade0-4f10-9908-3da1526301af_calibrate_898342_31.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898342/dc2c0f58-ade0-4f10-9908-3da1526301af_calibrate_898342_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:45:33 proc_lsst.multi:146 [INFO] found job 17008904 in provider astro proc_lsst.multi INFO: found job 17008904 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:45:35 proc_lsst.multi:146 [INFO] found job 17008905 in provider astro proc_lsst.multi INFO: found job 17008905 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 5 try 0 failed parsl.dataflow.dflow INFO: Task 5 marked for retry parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898326/e8dc36e3-899c-4e77-95b7-12cea144d25c_calibrate_898326_31.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898326/e8dc36e3-899c-4e77-95b7-12cea144d25c_calibrate_898326_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5 has memoization hash 7ab3a43ba4690db27a9d2acbf2b23156 parsl.dataflow.memoization INFO: Task 5 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b68ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5 try 1 launched on executor multi with executor id 158 parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898326/e8dc36e3-899c-4e77-95b7-12cea144d25c_calibrate_898326_31.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898326/e8dc36e3-899c-4e77-95b7-12cea144d25c_calibrate_898326_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:45:36 proc_lsst.multi:146 [INFO] found job 17008906 in provider astro proc_lsst.multi INFO: found job 17008906 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 13 try 0 failed parsl.dataflow.dflow INFO: Task 13 marked for retry parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898377/769a2947-4530-42bc-9032-1647e8b4baa3_calibrate_898377_31.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898377/769a2947-4530-42bc-9032-1647e8b4baa3_calibrate_898377_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 13 has memoization hash c72cc05cbe5d0f3f46a35f41069e2993 parsl.dataflow.memoization INFO: Task 13 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b69120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 13 try 1 launched on executor multi with executor id 159 parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898377/769a2947-4530-42bc-9032-1647e8b4baa3_calibrate_898377_31.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898377/769a2947-4530-42bc-9032-1647e8b4baa3_calibrate_898377_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:45:37 proc_lsst.multi:146 [INFO] found job 17008907 in provider astro proc_lsst.multi INFO: found job 17008907 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:45:39 proc_lsst.multi:146 [INFO] found job 17008908 in provider astro proc_lsst.multi INFO: found job 17008908 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:45:40 proc_lsst.multi:146 [INFO] found job 17008909 in provider astro proc_lsst.multi INFO: found job 17008909 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:45:41 proc_lsst.multi:146 [INFO] found job 17008910 in provider astro proc_lsst.multi INFO: found job 17008910 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 29 try 0 failed parsl.dataflow.dflow INFO: Task 29 marked for retry parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898320/4d762e6c-fe69-40c5-a31f-25c11aca98cb_calibrate_898320_31.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898320/4d762e6c-fe69-40c5-a31f-25c11aca98cb_calibrate_898320_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 29 has memoization hash 4039dedca7abdad7bd0d0159c6a786f7 parsl.dataflow.memoization INFO: Task 29 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b69bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 29 try 1 launched on executor multi with executor id 160 parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898320/4d762e6c-fe69-40c5-a31f-25c11aca98cb_calibrate_898320_31.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898320/4d762e6c-fe69-40c5-a31f-25c11aca98cb_calibrate_898320_31.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 12 try 0 failed parsl.dataflow.dflow INFO: Task 12 marked for retry parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898349/353d01b1-13b9-47ef-8df8-b3c7f7d84717_calibrate_898349_31.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898349/353d01b1-13b9-47ef-8df8-b3c7f7d84717_calibrate_898349_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 12 has memoization hash cad4018ac34f8c2db58928cbebfbe4e7 parsl.dataflow.memoization INFO: Task 12 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b68f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 12 try 1 launched on executor multi with executor id 161 parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898349/353d01b1-13b9-47ef-8df8-b3c7f7d84717_calibrate_898349_31.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898349/353d01b1-13b9-47ef-8df8-b3c7f7d84717_calibrate_898349_31.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 21 try 0 failed parsl.dataflow.dflow INFO: Task 21 marked for retry parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898353/c49c449d-4f7a-4ba2-91fa-945b7c653983_calibrate_898353_31.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898353/c49c449d-4f7a-4ba2-91fa-945b7c653983_calibrate_898353_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 21 has memoization hash 3c22e053a9a4ae24e4b55a94b408d45f parsl.dataflow.memoization INFO: Task 21 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b69620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 21 try 1 launched on executor multi with executor id 162 parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898353/c49c449d-4f7a-4ba2-91fa-945b7c653983_calibrate_898353_31.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898353/c49c449d-4f7a-4ba2-91fa-945b7c653983_calibrate_898353_31.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 27 try 0 failed parsl.dataflow.dflow INFO: Task 27 marked for retry parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898334/38bdbda3-4974-4bd8-b77c-ac6f04c45a29_calibrate_898334_31.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898334/38bdbda3-4974-4bd8-b77c-ac6f04c45a29_calibrate_898334_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 27 has memoization hash 2154875a4ca8ccfdcf0b441f8ebe8127 parsl.dataflow.memoization INFO: Task 27 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b69b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 27 try 1 launched on executor multi with executor id 163 parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898334/38bdbda3-4974-4bd8-b77c-ac6f04c45a29_calibrate_898334_31.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898334/38bdbda3-4974-4bd8-b77c-ac6f04c45a29_calibrate_898334_31.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 26 try 0 failed parsl.dataflow.dflow INFO: Task 26 marked for retry parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898374/b2ab386a-7074-4980-a0ae-626e602344c7_calibrate_898374_31.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898374/b2ab386a-7074-4980-a0ae-626e602344c7_calibrate_898374_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 26 has memoization hash 404a922ad98363cbbcd02ddc8c89da0a parsl.dataflow.memoization INFO: Task 26 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b698a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 26 try 1 launched on executor multi with executor id 164 parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898374/b2ab386a-7074-4980-a0ae-626e602344c7_calibrate_898374_31.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898374/b2ab386a-7074-4980-a0ae-626e602344c7_calibrate_898374_31.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 15 try 0 failed parsl.dataflow.dflow INFO: Task 15 marked for retry parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898341/b35cd10a-e0ae-4f4d-bded-3b07f4776216_calibrate_898341_31.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898341/b35cd10a-e0ae-4f4d-bded-3b07f4776216_calibrate_898341_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 15 has memoization hash 4302c68bf760c49455c325a400d545a9 parsl.dataflow.memoization INFO: Task 15 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b68ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 15 try 1 launched on executor multi with executor id 165 parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898341/b35cd10a-e0ae-4f4d-bded-3b07f4776216_calibrate_898341_31.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898341/b35cd10a-e0ae-4f4d-bded-3b07f4776216_calibrate_898341_31.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 28 try 0 failed parsl.dataflow.dflow INFO: Task 28 marked for retry parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898375/48301fc1-2890-4842-b21b-b4bda43648de_calibrate_898375_31.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898375/48301fc1-2890-4842-b21b-b4bda43648de_calibrate_898375_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash 9da04a8eed0e599dcf293179a3c308c4 parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b69c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 1 launched on executor multi with executor id 166 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898375/48301fc1-2890-4842-b21b-b4bda43648de_calibrate_898375_31.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898375/48301fc1-2890-4842-b21b-b4bda43648de_calibrate_898375_31.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 22 try 0 failed parsl.dataflow.dflow INFO: Task 22 marked for retry parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898366/78353f3c-57e1-4237-8cc1-01f5ac43e0ee_calibrate_898366_31.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898366/78353f3c-57e1-4237-8cc1-01f5ac43e0ee_calibrate_898366_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash 483dcf17c6bb676a0286f433a5d209d2 parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b69800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 1 launched on executor multi with executor id 167 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898366/78353f3c-57e1-4237-8cc1-01f5ac43e0ee_calibrate_898366_31.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898366/78353f3c-57e1-4237-8cc1-01f5ac43e0ee_calibrate_898366_31.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 20 try 0 failed parsl.dataflow.dflow INFO: Task 20 marked for retry parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898346/fa1671bf-7a1a-4e67-8fbe-bb109b12b349_calibrate_898346_31.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898346/fa1671bf-7a1a-4e67-8fbe-bb109b12b349_calibrate_898346_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 20 has memoization hash 68bf16d6705930d3552ef85355e36440 parsl.dataflow.memoization INFO: Task 20 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b696c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 20 try 1 launched on executor multi with executor id 168 parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898346/fa1671bf-7a1a-4e67-8fbe-bb109b12b349_calibrate_898346_31.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898346/fa1671bf-7a1a-4e67-8fbe-bb109b12b349_calibrate_898346_31.stderr parsl.channels.local.local WARNING: Execution of command 'squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934'' failed due to Command 'squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934'' timed out after 10 seconds parsl.utils ERROR: Callback threw an exception - logging and proceeding anyway Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/utils.py", line 341, in make_callback self.callback(*self.cb_args) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/job_status_poller.py", line 114, in poll self._update_state() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/job_status_poller.py", line 121, in _update_state item.poll(now) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/job_status_poller.py", line 47, in poll self._status = self._executor.status() ^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/executors/status_handling.py", line 109, in status status = self._make_status_dict(block_ids, self._provider.status(job_ids)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/gscratch/dirac/shared/opt/proc_lsst/python/proc_lsst/multi.py", line 190, in status statuses += self.providers[provider].status([job_id]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/providers/cluster_provider.py", line 142, in status self._status() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/providers/slurm/slurm.py", line 157, in _status retcode, stdout, stderr = self.execute_wait(cmd) ^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/providers/cluster_provider.py", line 83, in execute_wait return self.channel.execute_wait(cmd, t) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/channels/local/local.py", line 71, in execute_wait (stdout, stderr) = proc.communicate(timeout=walltime) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/subprocess.py", line 1209, in communicate stdout, stderr = self._communicate(input, endtime, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/subprocess.py", line 2114, in _communicate self._check_timeout(endtime, orig_timeout, stdout, stderr) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/subprocess.py", line 1253, in _check_timeout raise TimeoutExpired( subprocess.TimeoutExpired: Command 'squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934'' timed out after 10 seconds 2024-03-13 08:45:52 proc_lsst.multi:146 [INFO] found job 34092 in provider local proc_lsst.multi INFO: found job 34092 in provider local 2024-03-13 08:45:52 proc_lsst.multi:146 [INFO] found job 17008896 in provider astro proc_lsst.multi INFO: found job 17008896 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:45:53 proc_lsst.multi:146 [INFO] found job 17008898 in provider astro proc_lsst.multi INFO: found job 17008898 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:45:55 proc_lsst.multi:146 [INFO] found job 17008899 in provider astro proc_lsst.multi INFO: found job 17008899 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:45:56 proc_lsst.multi:146 [INFO] found job 17008900 in provider astro proc_lsst.multi INFO: found job 17008900 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 9 try 0 failed parsl.dataflow.dflow INFO: Task 9 marked for retry parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898382/4ab7025a-27b7-4b84-91f5-ad14420c245e_calibrate_898382_7.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898382/4ab7025a-27b7-4b84-91f5-ad14420c245e_calibrate_898382_7.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 9 has memoization hash 6bdcb389812a683b49c783b6c448d66b parsl.dataflow.memoization INFO: Task 9 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b68400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 9 try 1 launched on executor multi with executor id 169 parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898382/4ab7025a-27b7-4b84-91f5-ad14420c245e_calibrate_898382_7.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898382/4ab7025a-27b7-4b84-91f5-ad14420c245e_calibrate_898382_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:45:58 proc_lsst.multi:146 [INFO] found job 17008901 in provider astro proc_lsst.multi INFO: found job 17008901 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:45:59 proc_lsst.multi:146 [INFO] found job 17008902 in provider astro proc_lsst.multi INFO: found job 17008902 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:46:00 proc_lsst.multi:146 [INFO] found job 17008903 in provider astro proc_lsst.multi INFO: found job 17008903 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:46:02 proc_lsst.multi:146 [INFO] found job 17008904 in provider astro proc_lsst.multi INFO: found job 17008904 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:46:03 proc_lsst.multi:146 [INFO] found job 17008905 in provider astro proc_lsst.multi INFO: found job 17008905 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:46:05 proc_lsst.multi:146 [INFO] found job 17008906 in provider astro proc_lsst.multi INFO: found job 17008906 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:46:06 proc_lsst.multi:146 [INFO] found job 17008907 in provider astro proc_lsst.multi INFO: found job 17008907 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:46:07 proc_lsst.multi:146 [INFO] found job 17008908 in provider astro proc_lsst.multi INFO: found job 17008908 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:46:09 proc_lsst.multi:146 [INFO] found job 17008909 in provider astro proc_lsst.multi INFO: found job 17008909 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 34 try 0 failed parsl.dataflow.dflow INFO: Task 34 marked for retry parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898354/c244ec8e-369a-448b-a0ee-6e385648f4b1_calibrate_898354_31.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898354/c244ec8e-369a-448b-a0ee-6e385648f4b1_calibrate_898354_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash 5947a47f671ed559925e25b77189cefd parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b69ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 1 launched on executor multi with executor id 170 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898354/c244ec8e-369a-448b-a0ee-6e385648f4b1_calibrate_898354_31.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898354/c244ec8e-369a-448b-a0ee-6e385648f4b1_calibrate_898354_31.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 8 try 0 failed parsl.dataflow.dflow INFO: Task 8 marked for retry parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898366/3faa2151-682e-4b26-b2ce-a961451dc732_calibrate_898366_61.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898366/3faa2151-682e-4b26-b2ce-a961451dc732_calibrate_898366_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 8 has memoization hash 0ea846d7f41dde9c5463dfa6179c7af3 parsl.dataflow.memoization INFO: Task 8 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b68cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 8 try 1 launched on executor multi with executor id 171 parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898366/3faa2151-682e-4b26-b2ce-a961451dc732_calibrate_898366_61.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898366/3faa2151-682e-4b26-b2ce-a961451dc732_calibrate_898366_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:46:10 proc_lsst.multi:146 [INFO] found job 17008910 in provider astro proc_lsst.multi INFO: found job 17008910 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 2 try 0 failed parsl.dataflow.dflow INFO: Task 2 marked for retry parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898333/6df343bb-eaa6-4a48-a6cd-e1dd2bbc6bb7_calibrate_898333_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898333/6df343bb-eaa6-4a48-a6cd-e1dd2bbc6bb7_calibrate_898333_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2 has memoization hash 26209eddfa42689388c6f4e0bfdc4f38 parsl.dataflow.memoization INFO: Task 2 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b687c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2 try 1 launched on executor multi with executor id 172 parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898333/6df343bb-eaa6-4a48-a6cd-e1dd2bbc6bb7_calibrate_898333_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898333/6df343bb-eaa6-4a48-a6cd-e1dd2bbc6bb7_calibrate_898333_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:46:11 proc_lsst.multi:146 [INFO] found job 17008911 in provider astro proc_lsst.multi INFO: found job 17008911 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 43 try 0 failed parsl.dataflow.dflow INFO: Task 43 marked for retry parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898336/f5525908-304f-433e-9c72-7dddccdbfb5c_calibrate_898336_31.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898336/f5525908-304f-433e-9c72-7dddccdbfb5c_calibrate_898336_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 43 has memoization hash 68e95a36b0438dbfc00da1206a9ce4d5 parsl.dataflow.memoization INFO: Task 43 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 43 try 1 launched on executor multi with executor id 173 parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898336/f5525908-304f-433e-9c72-7dddccdbfb5c_calibrate_898336_31.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898336/f5525908-304f-433e-9c72-7dddccdbfb5c_calibrate_898336_31.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 49 try 0 failed parsl.dataflow.dflow INFO: Task 49 marked for retry parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898327/f0e8c5c2-1e4e-4b94-97cf-0e192997e8f0_calibrate_898327_31.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898327/f0e8c5c2-1e4e-4b94-97cf-0e192997e8f0_calibrate_898327_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 49 has memoization hash 61298ebe490fd5b5bcb32fbd1ea462ad parsl.dataflow.memoization INFO: Task 49 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 49 try 1 launched on executor multi with executor id 174 parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898327/f0e8c5c2-1e4e-4b94-97cf-0e192997e8f0_calibrate_898327_31.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898327/f0e8c5c2-1e4e-4b94-97cf-0e192997e8f0_calibrate_898327_31.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 44 try 0 failed parsl.dataflow.dflow INFO: Task 44 marked for retry parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898324/f7963f80-1c9c-4bb7-97aa-466c1c22bdb9_calibrate_898324_31.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898324/f7963f80-1c9c-4bb7-97aa-466c1c22bdb9_calibrate_898324_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 44 has memoization hash 4069964c40da3c18bb04d283d4cd2a2e parsl.dataflow.memoization INFO: Task 44 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 44 try 1 launched on executor multi with executor id 175 parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898324/f7963f80-1c9c-4bb7-97aa-466c1c22bdb9_calibrate_898324_31.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898324/f7963f80-1c9c-4bb7-97aa-466c1c22bdb9_calibrate_898324_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:46:13 proc_lsst.multi:146 [INFO] found job 17008913 in provider astro proc_lsst.multi INFO: found job 17008913 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:46:14 proc_lsst.multi:146 [INFO] found job 17008914 in provider astro proc_lsst.multi INFO: found job 17008914 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 35 try 0 failed parsl.dataflow.dflow INFO: Task 35 marked for retry parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898307/a34da88c-ce76-49a3-9a04-93e927f2d5fa_calibrate_898307_31.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898307/a34da88c-ce76-49a3-9a04-93e927f2d5fa_calibrate_898307_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 35 has memoization hash f7549c3b5ee5ec0cd39d9e522a784dd2 parsl.dataflow.memoization INFO: Task 35 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b69f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 35 try 1 launched on executor multi with executor id 176 parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898307/a34da88c-ce76-49a3-9a04-93e927f2d5fa_calibrate_898307_31.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898307/a34da88c-ce76-49a3-9a04-93e927f2d5fa_calibrate_898307_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:46:16 proc_lsst.multi:146 [INFO] found job 17008915 in provider astro proc_lsst.multi INFO: found job 17008915 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 56 try 0 failed parsl.dataflow.dflow INFO: Task 56 marked for retry parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898301/8ac197d9-bd72-461c-90c7-d994ec4e890c_characterizeImage_898301_2.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898301/8ac197d9-bd72-461c-90c7-d994ec4e890c_characterizeImage_898301_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 56 has memoization hash 8a5856ee7d0c61a4546900cabfbfe1c9 parsl.dataflow.memoization INFO: Task 56 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b694e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 56 try 1 launched on executor multi with executor id 177 parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898301/8ac197d9-bd72-461c-90c7-d994ec4e890c_characterizeImage_898301_2.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898301/8ac197d9-bd72-461c-90c7-d994ec4e890c_characterizeImage_898301_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 52 try 0 failed parsl.dataflow.dflow INFO: Task 52 marked for retry parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898352/3a68dde5-3364-44e6-9bda-ed56a9f9eaac_calibrate_898352_31.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898352/3a68dde5-3364-44e6-9bda-ed56a9f9eaac_calibrate_898352_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash 546bcd619bfed86dbc58546373cc23c9 parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 1 launched on executor multi with executor id 178 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898352/3a68dde5-3364-44e6-9bda-ed56a9f9eaac_calibrate_898352_31.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898352/3a68dde5-3364-44e6-9bda-ed56a9f9eaac_calibrate_898352_31.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 58 try 0 failed parsl.dataflow.dflow INFO: Task 58 marked for retry parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898311/72116209-a5e5-4f23-a885-45dbd5e314e6_characterizeImage_898311_2.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898311/72116209-a5e5-4f23-a885-45dbd5e314e6_characterizeImage_898311_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 58 has memoization hash 871f1c0bfc2b03edddb157476d0651fe parsl.dataflow.memoization INFO: Task 58 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 58 try 1 launched on executor multi with executor id 179 parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898311/72116209-a5e5-4f23-a885-45dbd5e314e6_characterizeImage_898311_2.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898311/72116209-a5e5-4f23-a885-45dbd5e314e6_characterizeImage_898311_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:46:17 proc_lsst.multi:146 [INFO] found job 17008917 in provider astro proc_lsst.multi INFO: found job 17008917 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 4 try 0 failed parsl.dataflow.dflow INFO: Task 4 marked for retry parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898375/21d6a845-9342-4d4b-907e-e59a7da8b7d0_calibrate_898375_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898375/21d6a845-9342-4d4b-907e-e59a7da8b7d0_calibrate_898375_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash 1b63349bb77c9e57be695c546ea22330 parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b68a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4 try 1 launched on executor multi with executor id 180 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898375/21d6a845-9342-4d4b-907e-e59a7da8b7d0_calibrate_898375_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898375/21d6a845-9342-4d4b-907e-e59a7da8b7d0_calibrate_898375_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:46:18 proc_lsst.multi:146 [INFO] found job 17008918 in provider astro proc_lsst.multi INFO: found job 17008918 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 62 try 0 failed parsl.dataflow.dflow INFO: Task 62 marked for retry parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898306/687414c1-d1e9-4f0b-9b67-b80b325c7bb7_characterizeImage_898306_2.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898306/687414c1-d1e9-4f0b-9b67-b80b325c7bb7_characterizeImage_898306_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 62 has memoization hash 2aba5b874159f2b9bd7bb6adc9fa55da parsl.dataflow.memoization INFO: Task 62 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 62 try 1 launched on executor multi with executor id 181 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898306/687414c1-d1e9-4f0b-9b67-b80b325c7bb7_characterizeImage_898306_2.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898306/687414c1-d1e9-4f0b-9b67-b80b325c7bb7_characterizeImage_898306_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 60 try 0 failed parsl.dataflow.dflow INFO: Task 60 marked for retry parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898290/ec58caa0-fe40-46e5-a79d-79858acbf8bc_characterizeImage_898290_2.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898290/ec58caa0-fe40-46e5-a79d-79858acbf8bc_characterizeImage_898290_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 60 has memoization hash 97a9946e5ae719d5aaff1e8b4d89c2e8 parsl.dataflow.memoization INFO: Task 60 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 60 try 1 launched on executor multi with executor id 182 parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898290/ec58caa0-fe40-46e5-a79d-79858acbf8bc_characterizeImage_898290_2.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898290/ec58caa0-fe40-46e5-a79d-79858acbf8bc_characterizeImage_898290_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 64 try 0 failed parsl.dataflow.dflow INFO: Task 64 marked for retry parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898380/90132901-dc69-42a9-86b3-bcd34dd462c2_characterizeImage_898380_2.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898380/90132901-dc69-42a9-86b3-bcd34dd462c2_characterizeImage_898380_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash cb65e91f672f3957fd9fc5594b209a2b parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 1 launched on executor multi with executor id 183 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898380/90132901-dc69-42a9-86b3-bcd34dd462c2_characterizeImage_898380_2.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898380/90132901-dc69-42a9-86b3-bcd34dd462c2_characterizeImage_898380_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:46:20 proc_lsst.multi:146 [INFO] found job 17008919 in provider astro proc_lsst.multi INFO: found job 17008919 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 14 try 0 failed parsl.dataflow.dflow INFO: Task 14 marked for retry parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898356/1070cfd5-8ce6-41fc-a039-2d4ac71300c0_calibrate_898356_61.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898356/1070cfd5-8ce6-41fc-a039-2d4ac71300c0_calibrate_898356_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 14 has memoization hash 976b0c75f578f168831d93d747f77b70 parsl.dataflow.memoization INFO: Task 14 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b69300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 14 try 1 launched on executor multi with executor id 184 parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898356/1070cfd5-8ce6-41fc-a039-2d4ac71300c0_calibrate_898356_61.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898356/1070cfd5-8ce6-41fc-a039-2d4ac71300c0_calibrate_898356_61.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 25 try 0 failed parsl.dataflow.dflow INFO: Task 25 marked for retry parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898319/61241879-2ac8-44d1-9098-dca5bb7dd64e_calibrate_898319_61.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898319/61241879-2ac8-44d1-9098-dca5bb7dd64e_calibrate_898319_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 25 has memoization hash 24efdbd9c833bcea3669feca1f99e441 parsl.dataflow.memoization INFO: Task 25 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b699e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 25 try 1 launched on executor multi with executor id 185 parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898319/61241879-2ac8-44d1-9098-dca5bb7dd64e_calibrate_898319_61.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898319/61241879-2ac8-44d1-9098-dca5bb7dd64e_calibrate_898319_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:46:21 proc_lsst.multi:146 [INFO] found job 17008921 in provider astro proc_lsst.multi INFO: found job 17008921 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:46:23 proc_lsst.multi:146 [INFO] found job 17008922 in provider astro proc_lsst.multi INFO: found job 17008922 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 42 try 0 failed parsl.dataflow.dflow INFO: Task 42 marked for retry parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898312/5cfe0bcb-2d52-4157-b2cd-bd5b702680b3_calibrate_898312_31.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898312/5cfe0bcb-2d52-4157-b2cd-bd5b702680b3_calibrate_898312_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 42 has memoization hash c4e1e65f38d0257accd555cc0a75ca7b parsl.dataflow.memoization INFO: Task 42 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6b2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 42 try 1 launched on executor multi with executor id 186 parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898312/5cfe0bcb-2d52-4157-b2cd-bd5b702680b3_calibrate_898312_31.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898312/5cfe0bcb-2d52-4157-b2cd-bd5b702680b3_calibrate_898312_31.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 72 try 0 failed parsl.dataflow.dflow INFO: Task 72 marked for retry parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898375/9bad3368-9d28-420c-bd65-b5390f0dd3b5_characterizeImage_898375_2.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898375/9bad3368-9d28-420c-bd65-b5390f0dd3b5_characterizeImage_898375_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 72 has memoization hash 5e1fcaaa00a419d8d4e76a5c2bd888c7 parsl.dataflow.memoization INFO: Task 72 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 72 try 1 launched on executor multi with executor id 187 parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898375/9bad3368-9d28-420c-bd65-b5390f0dd3b5_characterizeImage_898375_2.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898375/9bad3368-9d28-420c-bd65-b5390f0dd3b5_characterizeImage_898375_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 70 try 0 failed parsl.dataflow.dflow INFO: Task 70 marked for retry parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898288/256364ee-b666-43fa-9aae-b3d74aa5385a_characterizeImage_898288_2.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898288/256364ee-b666-43fa-9aae-b3d74aa5385a_characterizeImage_898288_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash 3b2ca9a94f256047962755948959ef06 parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 1 launched on executor multi with executor id 188 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898288/256364ee-b666-43fa-9aae-b3d74aa5385a_characterizeImage_898288_2.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898288/256364ee-b666-43fa-9aae-b3d74aa5385a_characterizeImage_898288_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 23 try 0 failed parsl.dataflow.dflow INFO: Task 23 marked for retry parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898345/5f136065-162d-49e3-9de7-31bf464a799c_calibrate_898345_61.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898345/5f136065-162d-49e3-9de7-31bf464a799c_calibrate_898345_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 23 has memoization hash 8144802c995d4528edd2552745882d99 parsl.dataflow.memoization INFO: Task 23 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b69940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 23 try 1 launched on executor multi with executor id 189 parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898345/5f136065-162d-49e3-9de7-31bf464a799c_calibrate_898345_61.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898345/5f136065-162d-49e3-9de7-31bf464a799c_calibrate_898345_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:46:24 proc_lsst.multi:146 [INFO] found job 17008923 in provider astro proc_lsst.multi INFO: found job 17008923 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 68 try 0 failed parsl.dataflow.dflow INFO: Task 68 marked for retry parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898334/cfea5b02-a172-4f43-a059-3065b6a16fe7_characterizeImage_898334_2.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898334/cfea5b02-a172-4f43-a059-3065b6a16fe7_characterizeImage_898334_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 68 has memoization hash 25e7d311cd1cde0309d3624f17af3185 parsl.dataflow.memoization INFO: Task 68 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 68 try 1 launched on executor multi with executor id 190 parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898334/cfea5b02-a172-4f43-a059-3065b6a16fe7_characterizeImage_898334_2.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898334/cfea5b02-a172-4f43-a059-3065b6a16fe7_characterizeImage_898334_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 6 try 0 failed parsl.dataflow.dflow INFO: Task 6 marked for retry parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898374/0eeb0a0c-dc47-435f-bb6a-652c42051f5c_calibrate_898374_61.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898374/0eeb0a0c-dc47-435f-bb6a-652c42051f5c_calibrate_898374_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 6 has memoization hash d97d0a7bdaf10f72d049ce95d7d29033 parsl.dataflow.memoization INFO: Task 6 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b3a3e1b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 6 try 1 launched on executor multi with executor id 191 parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898374/0eeb0a0c-dc47-435f-bb6a-652c42051f5c_calibrate_898374_61.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898374/0eeb0a0c-dc47-435f-bb6a-652c42051f5c_calibrate_898374_61.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 74 try 0 failed parsl.dataflow.dflow INFO: Task 74 marked for retry parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898326/7d3152b1-6692-4435-9366-c053e4873078_characterizeImage_898326_2.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898326/7d3152b1-6692-4435-9366-c053e4873078_characterizeImage_898326_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 74 has memoization hash 411d81f3ab7f15dad4bf228ec2a2a01a parsl.dataflow.memoization INFO: Task 74 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 74 try 1 launched on executor multi with executor id 192 parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898326/7d3152b1-6692-4435-9366-c053e4873078_characterizeImage_898326_2.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898326/7d3152b1-6692-4435-9366-c053e4873078_characterizeImage_898326_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:46:25 proc_lsst.multi:146 [INFO] found job 17008924 in provider astro proc_lsst.multi INFO: found job 17008924 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 78 try 0 failed parsl.dataflow.dflow INFO: Task 78 marked for retry parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898349/38416ffd-f62d-4e69-b703-f02832de1aea_characterizeImage_898349_2.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898349/38416ffd-f62d-4e69-b703-f02832de1aea_characterizeImage_898349_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 78 has memoization hash 9352761a2d2f84142e4de4fafc6a6e0e parsl.dataflow.memoization INFO: Task 78 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394ba8180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 78 try 1 launched on executor multi with executor id 193 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898349/38416ffd-f62d-4e69-b703-f02832de1aea_characterizeImage_898349_2.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898349/38416ffd-f62d-4e69-b703-f02832de1aea_characterizeImage_898349_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 76 try 0 failed parsl.dataflow.dflow INFO: Task 76 marked for retry parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898312/8c9728f4-fe0e-467c-9118-4c14df162c3d_characterizeImage_898312_2.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898312/8c9728f4-fe0e-467c-9118-4c14df162c3d_characterizeImage_898312_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash 3afd370b25175e2b9b252e3249fab9ea parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394ba80e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 1 launched on executor multi with executor id 194 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898312/8c9728f4-fe0e-467c-9118-4c14df162c3d_characterizeImage_898312_2.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898312/8c9728f4-fe0e-467c-9118-4c14df162c3d_characterizeImage_898312_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 18 try 0 failed parsl.dataflow.dflow INFO: Task 18 marked for retry parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898362/d41fc6e8-b2b1-467f-a720-63f17e62b88c_calibrate_898362_61.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898362/d41fc6e8-b2b1-467f-a720-63f17e62b88c_calibrate_898362_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 18 has memoization hash 7b65f5e5c021803b62472089700ad251 parsl.dataflow.memoization INFO: Task 18 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b69260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 18 try 1 launched on executor multi with executor id 195 parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898362/d41fc6e8-b2b1-467f-a720-63f17e62b88c_calibrate_898362_61.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898362/d41fc6e8-b2b1-467f-a720-63f17e62b88c_calibrate_898362_61.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 66 try 0 failed parsl.dataflow.dflow INFO: Task 66 marked for retry parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898355/10771c4c-8a1e-4e1e-9245-87f11f5ccd3e_characterizeImage_898355_2.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898355/10771c4c-8a1e-4e1e-9245-87f11f5ccd3e_characterizeImage_898355_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 66 has memoization hash 1c3e7bb0f533db41b0d33e340d0bfcc4 parsl.dataflow.memoization INFO: Task 66 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 66 try 1 launched on executor multi with executor id 196 parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898355/10771c4c-8a1e-4e1e-9245-87f11f5ccd3e_characterizeImage_898355_2.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898355/10771c4c-8a1e-4e1e-9245-87f11f5ccd3e_characterizeImage_898355_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 84 try 0 failed parsl.dataflow.dflow INFO: Task 84 marked for retry parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898323/5133c2f4-b9fe-4bf7-9ff3-9a5d3df60280_characterizeImage_898323_2.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898323/5133c2f4-b9fe-4bf7-9ff3-9a5d3df60280_characterizeImage_898323_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 84 has memoization hash 6c1c24bc55d30171015ad7bedde0f64d parsl.dataflow.memoization INFO: Task 84 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394ba8680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 84 try 1 launched on executor multi with executor id 197 parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898323/5133c2f4-b9fe-4bf7-9ff3-9a5d3df60280_characterizeImage_898323_2.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898323/5133c2f4-b9fe-4bf7-9ff3-9a5d3df60280_characterizeImage_898323_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 80 try 0 failed parsl.dataflow.dflow INFO: Task 80 marked for retry parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898317/f42869ae-e0a4-4695-a94d-eae09f51d55c_characterizeImage_898317_2.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898317/f42869ae-e0a4-4695-a94d-eae09f51d55c_characterizeImage_898317_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 80 has memoization hash 1493f5d7eb240fc798286fc52113f9e2 parsl.dataflow.memoization INFO: Task 80 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394ba82c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 80 try 1 launched on executor multi with executor id 198 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898317/f42869ae-e0a4-4695-a94d-eae09f51d55c_characterizeImage_898317_2.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898317/f42869ae-e0a4-4695-a94d-eae09f51d55c_characterizeImage_898317_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:46:27 proc_lsst.multi:146 [INFO] found job 17008925 in provider astro proc_lsst.multi INFO: found job 17008925 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 82 try 0 failed parsl.dataflow.dflow INFO: Task 82 marked for retry parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898358/e8ef6efb-7ecf-4b6d-a66f-16586850bca6_characterizeImage_898358_2.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898358/e8ef6efb-7ecf-4b6d-a66f-16586850bca6_characterizeImage_898358_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash 654acdcd0a0eda691b5063d512516b42 parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394ba84a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 1 launched on executor multi with executor id 199 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898358/e8ef6efb-7ecf-4b6d-a66f-16586850bca6_characterizeImage_898358_2.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898358/e8ef6efb-7ecf-4b6d-a66f-16586850bca6_characterizeImage_898358_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 10 try 0 failed parsl.dataflow.dflow INFO: Task 10 marked for retry parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898336/39bafb1f-6aae-4e48-9cf8-9bfc2976a642_calibrate_898336_61.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898336/39bafb1f-6aae-4e48-9cf8-9bfc2976a642_calibrate_898336_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash b5cabc075ea894567234827bf4b39aaa parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b68900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 10 try 1 launched on executor multi with executor id 200 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898336/39bafb1f-6aae-4e48-9cf8-9bfc2976a642_calibrate_898336_61.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898336/39bafb1f-6aae-4e48-9cf8-9bfc2976a642_calibrate_898336_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:46:28 proc_lsst.multi:146 [INFO] found job 17008926 in provider astro proc_lsst.multi INFO: found job 17008926 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 32 try 0 failed parsl.dataflow.dflow INFO: Task 32 marked for retry parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898351/f068b0a2-eb89-4a63-b361-e8d1e72e53c7_calibrate_898351_31.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898351/f068b0a2-eb89-4a63-b361-e8d1e72e53c7_calibrate_898351_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 32 has memoization hash ad83b57306beb23bf2701b4ec8428fe5 parsl.dataflow.memoization INFO: Task 32 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b69e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 32 try 1 launched on executor multi with executor id 201 parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898351/f068b0a2-eb89-4a63-b361-e8d1e72e53c7_calibrate_898351_31.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898351/f068b0a2-eb89-4a63-b361-e8d1e72e53c7_calibrate_898351_31.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 90 try 0 failed parsl.dataflow.dflow INFO: Task 90 marked for retry parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898321/b6b263c7-e416-493b-a529-87f8e2894e51_characterizeImage_898321_2.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898321/b6b263c7-e416-493b-a529-87f8e2894e51_characterizeImage_898321_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 90 has memoization hash 21c96c8f13668542a720b71501d6e6d6 parsl.dataflow.memoization INFO: Task 90 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394ba8c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 90 try 1 launched on executor multi with executor id 202 parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898321/b6b263c7-e416-493b-a529-87f8e2894e51_characterizeImage_898321_2.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898321/b6b263c7-e416-493b-a529-87f8e2894e51_characterizeImage_898321_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 88 try 0 failed parsl.dataflow.dflow INFO: Task 88 marked for retry parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898302/0b03baeb-44ab-4f8f-b76b-8fe5966ddbd8_characterizeImage_898302_2.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898302/0b03baeb-44ab-4f8f-b76b-8fe5966ddbd8_characterizeImage_898302_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 88 has memoization hash 628def22a10cce152316023db3117880 parsl.dataflow.memoization INFO: Task 88 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394ba8860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 88 try 1 launched on executor multi with executor id 203 parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898302/0b03baeb-44ab-4f8f-b76b-8fe5966ddbd8_characterizeImage_898302_2.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898302/0b03baeb-44ab-4f8f-b76b-8fe5966ddbd8_characterizeImage_898302_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:46:29 proc_lsst.multi:146 [INFO] found job 17008927 in provider astro proc_lsst.multi INFO: found job 17008927 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 92 try 0 failed parsl.dataflow.dflow INFO: Task 92 marked for retry parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898297/669fdf04-b21b-4b07-b670-2caae0e92c0d_characterizeImage_898297_2.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898297/669fdf04-b21b-4b07-b670-2caae0e92c0d_characterizeImage_898297_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash 261ae0ee6490810520ccbcac71c2312b parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394ba8e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 1 launched on executor multi with executor id 204 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898297/669fdf04-b21b-4b07-b670-2caae0e92c0d_characterizeImage_898297_2.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898297/669fdf04-b21b-4b07-b670-2caae0e92c0d_characterizeImage_898297_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 86 try 0 failed parsl.dataflow.dflow INFO: Task 86 marked for retry parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898352/ea28b3c9-e970-4b1f-9393-bd5bf5dd9520_characterizeImage_898352_2.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898352/ea28b3c9-e970-4b1f-9393-bd5bf5dd9520_characterizeImage_898352_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 86 has memoization hash 28c150fec6c19bf55aa1b4c8944de0f6 parsl.dataflow.memoization INFO: Task 86 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394ba8a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 86 try 1 launched on executor multi with executor id 205 parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898352/ea28b3c9-e970-4b1f-9393-bd5bf5dd9520_characterizeImage_898352_2.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898352/ea28b3c9-e970-4b1f-9393-bd5bf5dd9520_characterizeImage_898352_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:46:31 proc_lsst.multi:146 [INFO] found job 17008930 in provider astro proc_lsst.multi INFO: found job 17008930 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 54 try 0 failed parsl.dataflow.dflow INFO: Task 54 marked for retry parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898318/b13456ab-f9fa-4fb6-81b1-66a42b8b3c13_calibrate_898318_31.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898318/b13456ab-f9fa-4fb6-81b1-66a42b8b3c13_calibrate_898318_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 54 has memoization hash 71a530e2f28b0764e94bcbd4c6ecbd86 parsl.dataflow.memoization INFO: Task 54 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 54 try 1 launched on executor multi with executor id 206 parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898318/b13456ab-f9fa-4fb6-81b1-66a42b8b3c13_calibrate_898318_31.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898318/b13456ab-f9fa-4fb6-81b1-66a42b8b3c13_calibrate_898318_31.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 94 try 0 failed parsl.dataflow.dflow INFO: Task 94 marked for retry parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898332/b22b4ad9-5899-4a01-8fed-8564cd53ba37_characterizeImage_898332_2.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898332/b22b4ad9-5899-4a01-8fed-8564cd53ba37_characterizeImage_898332_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 94 has memoization hash f59722a9d505c1b91a9d5f565af0f069 parsl.dataflow.memoization INFO: Task 94 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394ba8fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 94 try 1 launched on executor multi with executor id 207 parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898332/b22b4ad9-5899-4a01-8fed-8564cd53ba37_characterizeImage_898332_2.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898332/b22b4ad9-5899-4a01-8fed-8564cd53ba37_characterizeImage_898332_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 102 try 0 failed parsl.dataflow.dflow INFO: Task 102 marked for retry parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898365/041b194e-fef4-41c1-9660-e78ee710ec76_characterizeImage_898365_2.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898365/041b194e-fef4-41c1-9660-e78ee710ec76_characterizeImage_898365_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 102 has memoization hash 64e7fc366a0525f9369946feb5abbf09 parsl.dataflow.memoization INFO: Task 102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394ba9760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 102 try 1 launched on executor multi with executor id 208 parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898365/041b194e-fef4-41c1-9660-e78ee710ec76_characterizeImage_898365_2.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898365/041b194e-fef4-41c1-9660-e78ee710ec76_characterizeImage_898365_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 100 try 0 failed parsl.dataflow.dflow INFO: Task 100 marked for retry parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898373/38ee9896-518b-48b2-a68f-de2b4b9536df_characterizeImage_898373_2.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898373/38ee9896-518b-48b2-a68f-de2b4b9536df_characterizeImage_898373_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash fd409323414c1a1d0f6cd6241a0ac7a1 parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394ba9580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 1 launched on executor multi with executor id 209 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898373/38ee9896-518b-48b2-a68f-de2b4b9536df_characterizeImage_898373_2.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898373/38ee9896-518b-48b2-a68f-de2b4b9536df_characterizeImage_898373_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:46:32 proc_lsst.multi:146 [INFO] found job 17008934 in provider astro proc_lsst.multi INFO: found job 17008934 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 96 try 0 failed parsl.dataflow.dflow INFO: Task 96 marked for retry parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898328/d604c1ba-966b-4f42-8457-f96d32e53430_characterizeImage_898328_2.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898328/d604c1ba-966b-4f42-8457-f96d32e53430_characterizeImage_898328_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 96 has memoization hash 4cd75d7ac7ccaa697b997bf5d33d8730 parsl.dataflow.memoization INFO: Task 96 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394ba93a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 96 try 1 launched on executor multi with executor id 210 parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898328/d604c1ba-966b-4f42-8457-f96d32e53430_characterizeImage_898328_2.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898328/d604c1ba-966b-4f42-8457-f96d32e53430_characterizeImage_898328_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 98 try 0 failed parsl.dataflow.dflow INFO: Task 98 marked for retry parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898346/8b7896f6-d177-4d6d-977b-67c894156eee_characterizeImage_898346_2.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898346/8b7896f6-d177-4d6d-977b-67c894156eee_characterizeImage_898346_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 98 has memoization hash 4e8b658cbaab8e531dc6f1e523d932e2 parsl.dataflow.memoization INFO: Task 98 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394ba91c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 98 try 1 launched on executor multi with executor id 211 parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898346/8b7896f6-d177-4d6d-977b-67c894156eee_characterizeImage_898346_2.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898346/8b7896f6-d177-4d6d-977b-67c894156eee_characterizeImage_898346_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 108 try 0 failed parsl.dataflow.dflow INFO: Task 108 marked for retry parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898330/60a380f1-e740-4766-baed-0adb4a25953e_characterizeImage_898330_2.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898330/60a380f1-e740-4766-baed-0adb4a25953e_characterizeImage_898330_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 108 has memoization hash 004a77f79aec5d3f5723722d59e73a05 parsl.dataflow.memoization INFO: Task 108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394ba9b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 108 try 1 launched on executor multi with executor id 212 parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898330/60a380f1-e740-4766-baed-0adb4a25953e_characterizeImage_898330_2.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898330/60a380f1-e740-4766-baed-0adb4a25953e_characterizeImage_898330_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 104 try 0 failed parsl.dataflow.dflow INFO: Task 104 marked for retry parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898370/56990dd5-5747-4b9a-92a7-2fefb7fcd19b_characterizeImage_898370_2.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898370/56990dd5-5747-4b9a-92a7-2fefb7fcd19b_characterizeImage_898370_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 104 has memoization hash f00b1dd700441f6be9a0dd3f6d51e9a3 parsl.dataflow.memoization INFO: Task 104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394ba9940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 104 try 1 launched on executor multi with executor id 213 parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898370/56990dd5-5747-4b9a-92a7-2fefb7fcd19b_characterizeImage_898370_2.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898370/56990dd5-5747-4b9a-92a7-2fefb7fcd19b_characterizeImage_898370_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 16 try 0 failed parsl.dataflow.dflow INFO: Task 16 marked for retry parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898363/9a3d4b9b-637b-4d54-9756-08db927791ea_calibrate_898363_61.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898363/9a3d4b9b-637b-4d54-9756-08db927791ea_calibrate_898363_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash 4b3acee8a57858b7460670ddb218b574 parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b693a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 16 try 1 launched on executor multi with executor id 214 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898363/9a3d4b9b-637b-4d54-9756-08db927791ea_calibrate_898363_61.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898363/9a3d4b9b-637b-4d54-9756-08db927791ea_calibrate_898363_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:46:33 proc_lsst.multi:146 [INFO] found job 17008943 in provider escience_rtx6k proc_lsst.multi INFO: found job 17008943 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008943,17008949,17008950,17008951,17008952' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 110 try 0 failed parsl.dataflow.dflow INFO: Task 110 marked for retry parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898324/b42216a1-ed0d-4817-b690-a87e5f1b15a6_characterizeImage_898324_2.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898324/b42216a1-ed0d-4817-b690-a87e5f1b15a6_characterizeImage_898324_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 110 has memoization hash bc1139a2902a49b0759292637d4b2394 parsl.dataflow.memoization INFO: Task 110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394ba9ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 110 try 1 launched on executor multi with executor id 215 parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898324/b42216a1-ed0d-4817-b690-a87e5f1b15a6_characterizeImage_898324_2.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898324/b42216a1-ed0d-4817-b690-a87e5f1b15a6_characterizeImage_898324_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 112 try 0 failed parsl.dataflow.dflow INFO: Task 112 marked for retry parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898322/a639d512-32dc-4105-9ea8-df8022357a3b_characterizeImage_898322_2.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898322/a639d512-32dc-4105-9ea8-df8022357a3b_characterizeImage_898322_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 112 has memoization hash 681c26047d26280295a0ebbee5ec6622 parsl.dataflow.memoization INFO: Task 112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394baa0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 112 try 1 launched on executor multi with executor id 216 parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898322/a639d512-32dc-4105-9ea8-df8022357a3b_characterizeImage_898322_2.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898322/a639d512-32dc-4105-9ea8-df8022357a3b_characterizeImage_898322_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 114 try 0 failed parsl.dataflow.dflow INFO: Task 114 marked for retry parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898303/a533e21c-7e5e-45ca-aee7-0090ab76421b_characterizeImage_898303_2.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898303/a533e21c-7e5e-45ca-aee7-0090ab76421b_characterizeImage_898303_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 114 has memoization hash d714e2248c9ad715ae07492a0747914d parsl.dataflow.memoization INFO: Task 114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394baa2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 114 try 1 launched on executor multi with executor id 217 parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898303/a533e21c-7e5e-45ca-aee7-0090ab76421b_characterizeImage_898303_2.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898303/a533e21c-7e5e-45ca-aee7-0090ab76421b_characterizeImage_898303_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008950 R 17008951 R 17008952 R 17008949 R 17008943 R parsl.providers.slurm.slurm DEBUG: Updating job 17008950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008943 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:46:35 proc_lsst.multi:146 [INFO] found job 17008949 in provider escience_rtx6k proc_lsst.multi INFO: found job 17008949 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008943,17008949,17008950,17008951,17008952' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 118 try 0 failed parsl.dataflow.dflow INFO: Task 118 marked for retry parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898345/cbf6c8ea-e38c-4321-85ca-0eac41820570_characterizeImage_898345_2.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898345/cbf6c8ea-e38c-4321-85ca-0eac41820570_characterizeImage_898345_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash 649fc220bc2bffba313897db8374eb97 parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394baa480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 1 launched on executor multi with executor id 218 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898345/cbf6c8ea-e38c-4321-85ca-0eac41820570_characterizeImage_898345_2.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898345/cbf6c8ea-e38c-4321-85ca-0eac41820570_characterizeImage_898345_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 120 try 0 failed parsl.dataflow.dflow INFO: Task 120 marked for retry parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898369/2fe593b4-90ae-4c32-b322-54407729f112_characterizeImage_898369_2.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898369/2fe593b4-90ae-4c32-b322-54407729f112_characterizeImage_898369_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 120 has memoization hash 4f1959275519093a86082fc455f48e67 parsl.dataflow.memoization INFO: Task 120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394baa840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 120 try 1 launched on executor multi with executor id 219 parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898369/2fe593b4-90ae-4c32-b322-54407729f112_characterizeImage_898369_2.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898369/2fe593b4-90ae-4c32-b322-54407729f112_characterizeImage_898369_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 122 try 0 failed parsl.dataflow.dflow INFO: Task 122 marked for retry parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898310/f0d0662d-41f6-4456-a487-ef4267af8fca_characterizeImage_898310_2.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898310/f0d0662d-41f6-4456-a487-ef4267af8fca_characterizeImage_898310_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 122 has memoization hash a5f8477f44414355d2876740a66b990a parsl.dataflow.memoization INFO: Task 122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394baaa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 122 try 1 launched on executor multi with executor id 220 parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898310/f0d0662d-41f6-4456-a487-ef4267af8fca_characterizeImage_898310_2.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898310/f0d0662d-41f6-4456-a487-ef4267af8fca_characterizeImage_898310_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 40 try 0 failed parsl.dataflow.dflow INFO: Task 40 marked for retry parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898334/39685529-733b-419f-b862-f7a086c8ce6a_calibrate_898334_36.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898334/39685529-733b-419f-b862-f7a086c8ce6a_calibrate_898334_36.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash c7f9ef97f36aad0c3ad3f136aee93c45 parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 40 try 1 launched on executor multi with executor id 221 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898334/39685529-733b-419f-b862-f7a086c8ce6a_calibrate_898334_36.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898334/39685529-733b-419f-b862-f7a086c8ce6a_calibrate_898334_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008950 R 17008951 R 17008952 R 17008949 R 17008943 R parsl.providers.slurm.slurm DEBUG: Updating job 17008950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008943 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:46:36 proc_lsst.multi:146 [INFO] found job 17008950 in provider escience_rtx6k proc_lsst.multi INFO: found job 17008950 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008943,17008949,17008950,17008951,17008952' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 106 try 0 failed parsl.dataflow.dflow INFO: Task 106 marked for retry parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898356/1dde0e09-6994-4851-bffb-41645b1063f6_characterizeImage_898356_2.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898356/1dde0e09-6994-4851-bffb-41645b1063f6_characterizeImage_898356_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash 10e68e6a2238ca602f91fc1a6083824b parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394ba9d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 106 try 1 launched on executor multi with executor id 222 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898356/1dde0e09-6994-4851-bffb-41645b1063f6_characterizeImage_898356_2.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898356/1dde0e09-6994-4851-bffb-41645b1063f6_characterizeImage_898356_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 124 try 0 failed parsl.dataflow.dflow INFO: Task 124 marked for retry parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898327/78d3fffa-c228-475d-941c-2f6d356c99b2_characterizeImage_898327_2.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898327/78d3fffa-c228-475d-941c-2f6d356c99b2_characterizeImage_898327_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 124 has memoization hash d2ee61bdc2b93667ff095c09f615fac2 parsl.dataflow.memoization INFO: Task 124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394baac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 124 try 1 launched on executor multi with executor id 223 parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898327/78d3fffa-c228-475d-941c-2f6d356c99b2_characterizeImage_898327_2.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898327/78d3fffa-c228-475d-941c-2f6d356c99b2_characterizeImage_898327_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008950 R 17008951 R 17008952 R 17008949 R 17008943 R parsl.providers.slurm.slurm DEBUG: Updating job 17008950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008943 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:46:37 proc_lsst.multi:146 [INFO] found job 17008951 in provider escience_rtx6k proc_lsst.multi INFO: found job 17008951 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008943,17008949,17008950,17008951,17008952' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 130 try 0 failed parsl.dataflow.dflow INFO: Task 130 marked for retry parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898342/e6cece1d-a943-4e40-ae51-8391141a889a_characterizeImage_898342_2.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898342/e6cece1d-a943-4e40-ae51-8391141a889a_characterizeImage_898342_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 130 has memoization hash e2d9b4af3199bf1a4af8b80b93803cbb parsl.dataflow.memoization INFO: Task 130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bab1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 130 try 1 launched on executor multi with executor id 224 parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898342/e6cece1d-a943-4e40-ae51-8391141a889a_characterizeImage_898342_2.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898342/e6cece1d-a943-4e40-ae51-8391141a889a_characterizeImage_898342_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 132 try 0 failed parsl.dataflow.dflow INFO: Task 132 marked for retry parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898333/14edeee9-1368-4638-9bd8-cad4be9f250f_characterizeImage_898333_2.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898333/14edeee9-1368-4638-9bd8-cad4be9f250f_characterizeImage_898333_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 132 has memoization hash 46027d2f5a228a5f097bec7cb9256df5 parsl.dataflow.memoization INFO: Task 132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bab380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 132 try 1 launched on executor multi with executor id 225 parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898333/14edeee9-1368-4638-9bd8-cad4be9f250f_characterizeImage_898333_2.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898333/14edeee9-1368-4638-9bd8-cad4be9f250f_characterizeImage_898333_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 126 try 0 failed parsl.dataflow.dflow INFO: Task 126 marked for retry parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898295/dfde6b77-0194-4de1-b802-86a134975378_characterizeImage_898295_2.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898295/dfde6b77-0194-4de1-b802-86a134975378_characterizeImage_898295_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 126 has memoization hash b38f6a39639aa9b0ee832c766689c8fa parsl.dataflow.memoization INFO: Task 126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394baafc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 126 try 1 launched on executor multi with executor id 226 parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898295/dfde6b77-0194-4de1-b802-86a134975378_characterizeImage_898295_2.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898295/dfde6b77-0194-4de1-b802-86a134975378_characterizeImage_898295_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 116 try 0 failed parsl.dataflow.dflow INFO: Task 116 marked for retry parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898371/6677da51-c7e3-4e99-a0d2-010a7e7f57b9_characterizeImage_898371_2.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898371/6677da51-c7e3-4e99-a0d2-010a7e7f57b9_characterizeImage_898371_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 116 has memoization hash 66d86101b452de8bf87175fc34ff50a4 parsl.dataflow.memoization INFO: Task 116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394baa660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 116 try 1 launched on executor multi with executor id 227 parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898371/6677da51-c7e3-4e99-a0d2-010a7e7f57b9_characterizeImage_898371_2.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898371/6677da51-c7e3-4e99-a0d2-010a7e7f57b9_characterizeImage_898371_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 128 try 0 failed parsl.dataflow.dflow INFO: Task 128 marked for retry parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898376/b14bc756-1590-4a0e-b075-74eecaae062f_characterizeImage_898376_2.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898376/b14bc756-1590-4a0e-b075-74eecaae062f_characterizeImage_898376_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 128 has memoization hash e714a8c3f307643a6b4d1d75d6a45e73 parsl.dataflow.memoization INFO: Task 128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394baade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 128 try 1 launched on executor multi with executor id 228 parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898376/b14bc756-1590-4a0e-b075-74eecaae062f_characterizeImage_898376_2.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898376/b14bc756-1590-4a0e-b075-74eecaae062f_characterizeImage_898376_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 138 try 0 failed parsl.dataflow.dflow INFO: Task 138 marked for retry parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898339/91647e66-469e-416e-a268-f8eed54c914e_characterizeImage_898339_2.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898339/91647e66-469e-416e-a268-f8eed54c914e_characterizeImage_898339_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 138 has memoization hash ff5b909ecfc00a8ede54916c785b683f parsl.dataflow.memoization INFO: Task 138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bab740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 138 try 1 launched on executor multi with executor id 229 parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898339/91647e66-469e-416e-a268-f8eed54c914e_characterizeImage_898339_2.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898339/91647e66-469e-416e-a268-f8eed54c914e_characterizeImage_898339_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008950 R 17008951 R 17008952 R 17008949 R 17008943 R parsl.providers.slurm.slurm DEBUG: Updating job 17008950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008943 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:46:39 proc_lsst.multi:146 [INFO] found job 17008952 in provider escience_rtx6k proc_lsst.multi INFO: found job 17008952 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008943,17008949,17008950,17008951,17008952' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 19 try 0 failed parsl.dataflow.dflow INFO: Task 19 marked for retry parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898335/9ee28e39-dd1f-40cc-91e9-458e31ad9e35_calibrate_898335_61.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898335/9ee28e39-dd1f-40cc-91e9-458e31ad9e35_calibrate_898335_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 19 has memoization hash 2dd5cc0de2690a25ddf96527927bdc84 parsl.dataflow.memoization INFO: Task 19 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b69440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 19 try 1 launched on executor multi with executor id 230 parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898335/9ee28e39-dd1f-40cc-91e9-458e31ad9e35_calibrate_898335_61.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898335/9ee28e39-dd1f-40cc-91e9-458e31ad9e35_calibrate_898335_61.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 134 try 0 failed parsl.dataflow.dflow INFO: Task 134 marked for retry parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898293/3d3968e4-c012-4124-a509-f38e1e0d9fa8_characterizeImage_898293_2.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898293/3d3968e4-c012-4124-a509-f38e1e0d9fa8_characterizeImage_898293_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 134 has memoization hash eef3adc3e008b9c140003faf6a90c2b7 parsl.dataflow.memoization INFO: Task 134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bab560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 134 try 1 launched on executor multi with executor id 231 parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898293/3d3968e4-c012-4124-a509-f38e1e0d9fa8_characterizeImage_898293_2.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898293/3d3968e4-c012-4124-a509-f38e1e0d9fa8_characterizeImage_898293_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 11 try 0 failed parsl.dataflow.dflow INFO: Task 11 marked for retry parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898342/cf783a12-8612-47c1-ad91-69a33e09a85c_calibrate_898342_61.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898342/cf783a12-8612-47c1-ad91-69a33e09a85c_calibrate_898342_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 11 has memoization hash a02de1cca403ca71bfd1ce8526d5b68a parsl.dataflow.memoization INFO: Task 11 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b68c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 11 try 1 launched on executor multi with executor id 232 parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898342/cf783a12-8612-47c1-ad91-69a33e09a85c_calibrate_898342_61.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898342/cf783a12-8612-47c1-ad91-69a33e09a85c_calibrate_898342_61.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 136 try 0 failed parsl.dataflow.dflow INFO: Task 136 marked for retry parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898319/97c032f7-fed0-48f5-98c5-8ef97841e90c_characterizeImage_898319_2.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898319/97c032f7-fed0-48f5-98c5-8ef97841e90c_characterizeImage_898319_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 136 has memoization hash a4b8335b5d19157dd964fbb1e42fd80a parsl.dataflow.memoization INFO: Task 136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bab920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 136 try 1 launched on executor multi with executor id 233 parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898319/97c032f7-fed0-48f5-98c5-8ef97841e90c_characterizeImage_898319_2.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898319/97c032f7-fed0-48f5-98c5-8ef97841e90c_characterizeImage_898319_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008950 R 17008951 R 17008952 R 17008949 R 17008943 R parsl.providers.slurm.slurm DEBUG: Updating job 17008950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008943 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:46:40 proc_lsst.multi:146 [INFO] found job 17008953 in provider escience_a40 proc_lsst.multi INFO: found job 17008953 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008953,17008954' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 140 try 0 failed parsl.dataflow.dflow INFO: Task 140 marked for retry parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898318/fc5dc045-bf0e-43d6-95e3-1b7b0a0635f7_characterizeImage_898318_2.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898318/fc5dc045-bf0e-43d6-95e3-1b7b0a0635f7_characterizeImage_898318_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 140 has memoization hash a537e402d903b036ed90a11e2e1b344e parsl.dataflow.memoization INFO: Task 140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394babb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 140 try 1 launched on executor multi with executor id 234 parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898318/fc5dc045-bf0e-43d6-95e3-1b7b0a0635f7_characterizeImage_898318_2.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898318/fc5dc045-bf0e-43d6-95e3-1b7b0a0635f7_characterizeImage_898318_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 142 try 0 failed parsl.dataflow.dflow INFO: Task 142 marked for retry parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898287/3fded289-8eed-4ce8-b353-f1e8205743a1_characterizeImage_898287_2.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898287/3fded289-8eed-4ce8-b353-f1e8205743a1_characterizeImage_898287_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 142 has memoization hash c3c2e67415f37b774d20ed2a974e664b parsl.dataflow.memoization INFO: Task 142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394babce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 142 try 1 launched on executor multi with executor id 235 parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898287/3fded289-8eed-4ce8-b353-f1e8205743a1_characterizeImage_898287_2.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898287/3fded289-8eed-4ce8-b353-f1e8205743a1_characterizeImage_898287_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 144 try 0 failed parsl.dataflow.dflow INFO: Task 144 marked for retry parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898337/3bedf328-bece-48f2-a146-6fdd66b27359_characterizeImage_898337_2.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898337/3bedf328-bece-48f2-a146-6fdd66b27359_characterizeImage_898337_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 144 has memoization hash 9c97e3fa9e28b23d513b63c136164642 parsl.dataflow.memoization INFO: Task 144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bd8040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 144 try 1 launched on executor multi with executor id 236 parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898337/3bedf328-bece-48f2-a146-6fdd66b27359_characterizeImage_898337_2.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898337/3bedf328-bece-48f2-a146-6fdd66b27359_characterizeImage_898337_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 150 try 0 failed parsl.dataflow.dflow INFO: Task 150 marked for retry parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898344/ad25b03a-d309-4099-a4c8-f0db38b05219_characterizeImage_898344_2.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898344/ad25b03a-d309-4099-a4c8-f0db38b05219_characterizeImage_898344_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 150 has memoization hash e0f7a4f2a8ae6516d8b754d10ea1c74d parsl.dataflow.memoization INFO: Task 150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bd84a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 150 try 1 launched on executor multi with executor id 237 parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898344/ad25b03a-d309-4099-a4c8-f0db38b05219_characterizeImage_898344_2.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898344/ad25b03a-d309-4099-a4c8-f0db38b05219_characterizeImage_898344_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 152 try 0 failed parsl.dataflow.dflow INFO: Task 152 marked for retry parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898354/0704352b-dd37-4eb0-b3ca-3928ab62298f_characterizeImage_898354_2.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898354/0704352b-dd37-4eb0-b3ca-3928ab62298f_characterizeImage_898354_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 152 has memoization hash 407875da43491a938cd49d59b18c6273 parsl.dataflow.memoization INFO: Task 152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bd8680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 152 try 1 launched on executor multi with executor id 238 parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898354/0704352b-dd37-4eb0-b3ca-3928ab62298f_characterizeImage_898354_2.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898354/0704352b-dd37-4eb0-b3ca-3928ab62298f_characterizeImage_898354_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008954 R 17008953 R parsl.providers.slurm.slurm DEBUG: Updating job 17008954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008953 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:46:41 proc_lsst.multi:146 [INFO] found job 17008954 in provider escience_a40 proc_lsst.multi INFO: found job 17008954 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008953,17008954' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 146 try 0 failed parsl.dataflow.dflow INFO: Task 146 marked for retry parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898362/273324db-4d70-46d8-a1a7-f4b198e1783f_characterizeImage_898362_2.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898362/273324db-4d70-46d8-a1a7-f4b198e1783f_characterizeImage_898362_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 146 has memoization hash 2b0511c6c1b4436dd88b008f34f8d4a2 parsl.dataflow.memoization INFO: Task 146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bd82c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 146 try 1 launched on executor multi with executor id 239 parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898362/273324db-4d70-46d8-a1a7-f4b198e1783f_characterizeImage_898362_2.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898362/273324db-4d70-46d8-a1a7-f4b198e1783f_characterizeImage_898362_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 154 try 0 failed parsl.dataflow.dflow INFO: Task 154 marked for retry parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898294/f5b9f324-1334-4e2e-a191-f3b12aaa5965_characterizeImage_898294_2.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898294/f5b9f324-1334-4e2e-a191-f3b12aaa5965_characterizeImage_898294_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 154 has memoization hash eb57123a91a4dde1df4fd39a8b6f90d8 parsl.dataflow.memoization INFO: Task 154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bd8860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 154 try 1 launched on executor multi with executor id 240 parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898294/f5b9f324-1334-4e2e-a191-f3b12aaa5965_characterizeImage_898294_2.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898294/f5b9f324-1334-4e2e-a191-f3b12aaa5965_characterizeImage_898294_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 55 try 0 failed parsl.dataflow.dflow INFO: Task 55 marked for retry parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898331/9940692a-d823-4c18-bb68-1fb4b229ad0f_calibrate_898331_31.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898331/9940692a-d823-4c18-bb68-1fb4b229ad0f_calibrate_898331_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 55 has memoization hash e093fa6e2ddd0c7d3c73293fa8d0a1cf parsl.dataflow.memoization INFO: Task 55 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 55 try 1 launched on executor multi with executor id 241 parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898331/9940692a-d823-4c18-bb68-1fb4b229ad0f_calibrate_898331_31.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898331/9940692a-d823-4c18-bb68-1fb4b229ad0f_calibrate_898331_31.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 148 try 0 failed parsl.dataflow.dflow INFO: Task 148 marked for retry parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898360/6f8c9e81-9a6b-4554-8d15-ce766f7c0e51_characterizeImage_898360_2.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898360/6f8c9e81-9a6b-4554-8d15-ce766f7c0e51_characterizeImage_898360_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 148 has memoization hash 76e251355064399dc11343e930b21e23 parsl.dataflow.memoization INFO: Task 148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bd80e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 148 try 1 launched on executor multi with executor id 242 parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898360/6f8c9e81-9a6b-4554-8d15-ce766f7c0e51_characterizeImage_898360_2.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898360/6f8c9e81-9a6b-4554-8d15-ce766f7c0e51_characterizeImage_898360_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 30 try 0 failed parsl.dataflow.dflow INFO: Task 30 marked for retry parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898334/b995a848-ee4d-4405-a8c0-dda0691b96b0_calibrate_898334_61.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898334/b995a848-ee4d-4405-a8c0-dda0691b96b0_calibrate_898334_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 30 has memoization hash 6fd2140ec551b4a45f963c0c020e2ebf parsl.dataflow.memoization INFO: Task 30 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b69a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 30 try 1 launched on executor multi with executor id 243 parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898334/b995a848-ee4d-4405-a8c0-dda0691b96b0_calibrate_898334_61.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898334/b995a848-ee4d-4405-a8c0-dda0691b96b0_calibrate_898334_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008954 R 17008953 R parsl.providers.slurm.slurm DEBUG: Updating job 17008954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008953 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 38, active_tasks = 153 parsl.dataflow.strategy DEBUG: Executor multi has 153 active tasks, 38/0 running/pending blocks, and 39 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 38 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 1 more blocks parsl.executors.status_handling INFO: Scaling out by 1 blocks parsl.executors.status_handling INFO: Allocated block ID 38 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-38 2024-03-13 08:46:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 2024-03-13 08:46:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 08:46:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 08:46:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 08:46:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:46:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 08:46:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 08:46:43 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:46:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 08:46:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 08:46:43 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 08:46:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-13 08:46:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 08:46:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,198.48.92.26,n3009,10.64.65.9,10.64.129.9,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54132 --result_port=54008 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 156 try 0 failed parsl.dataflow.dflow INFO: Task 156 marked for retry parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898286/298e4d3d-7adc-4228-b469-702d4902d5d1_characterizeImage_898286_2.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898286/298e4d3d-7adc-4228-b469-702d4902d5d1_characterizeImage_898286_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 156 has memoization hash b67bc97637cec3612a5869fcc9cf54f0 parsl.dataflow.memoization INFO: Task 156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bd8c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 156 try 1 launched on executor multi with executor id 244 parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898286/298e4d3d-7adc-4228-b469-702d4902d5d1_characterizeImage_898286_2.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898286/298e4d3d-7adc-4228-b469-702d4902d5d1_characterizeImage_898286_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 17 try 0 failed parsl.dataflow.dflow INFO: Task 17 marked for retry parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898369/53ab6566-4b30-4473-a020-ede08ff4c708_calibrate_898369_61.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898369/53ab6566-4b30-4473-a020-ede08ff4c708_calibrate_898369_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 17 has memoization hash c3be2f6c16e1ce750d1a80c216b86dbf parsl.dataflow.memoization INFO: Task 17 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b69580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 17 try 1 launched on executor multi with executor id 245 parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898369/53ab6566-4b30-4473-a020-ede08ff4c708_calibrate_898369_61.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898369/53ab6566-4b30-4473-a020-ede08ff4c708_calibrate_898369_61.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 164 try 0 failed parsl.dataflow.dflow INFO: Task 164 marked for retry parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898350/6e8de21e-cc46-48e9-babd-fad71dbccbc4_characterizeImage_898350_2.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898350/6e8de21e-cc46-48e9-babd-fad71dbccbc4_characterizeImage_898350_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 164 has memoization hash 308fe0cd5112c0272d33c1f37c2dfe89 parsl.dataflow.memoization INFO: Task 164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bd91c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 164 try 1 launched on executor multi with executor id 246 parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898350/6e8de21e-cc46-48e9-babd-fad71dbccbc4_characterizeImage_898350_2.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898350/6e8de21e-cc46-48e9-babd-fad71dbccbc4_characterizeImage_898350_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 162 try 0 failed parsl.dataflow.dflow INFO: Task 162 marked for retry parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898315/e2b2c7ea-f52d-42e1-a0fd-3e90ac60d7e2_characterizeImage_898315_2.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898315/e2b2c7ea-f52d-42e1-a0fd-3e90ac60d7e2_characterizeImage_898315_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 162 has memoization hash ea6a56b7ace35c002723f8d6242d2385 parsl.dataflow.memoization INFO: Task 162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bd8fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 162 try 1 launched on executor multi with executor id 247 parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898315/e2b2c7ea-f52d-42e1-a0fd-3e90ac60d7e2_characterizeImage_898315_2.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898315/e2b2c7ea-f52d-42e1-a0fd-3e90ac60d7e2_characterizeImage_898315_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 31 try 0 failed parsl.dataflow.dflow INFO: Task 31 marked for retry parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898343/64d0e1db-ccf1-458e-bbd9-d102c90d7e8b_calibrate_898343_61.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898343/64d0e1db-ccf1-458e-bbd9-d102c90d7e8b_calibrate_898343_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 31 has memoization hash bcfe0a168ab99bf456c8b1f0cdfa7d07 parsl.dataflow.memoization INFO: Task 31 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b69da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 31 try 1 launched on executor multi with executor id 248 parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898343/64d0e1db-ccf1-458e-bbd9-d102c90d7e8b_calibrate_898343_61.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898343/64d0e1db-ccf1-458e-bbd9-d102c90d7e8b_calibrate_898343_61.stderr 2024-03-13 08:46:44 proc_lsst.multi:170 [INFO] [multi] job_id 17008961 proc_lsst.multi INFO: [multi] job_id 17008961 2024-03-13 08:46:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-13 08:46:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 08:46:44 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17008961 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17008961 parsl.executors.status_handling DEBUG: Launched block 38 on executor multi with job ID 17008961 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22761558686992 2024-03-13 08:46:44 proc_lsst.multi:146 [INFO] found job 34092 in provider local proc_lsst.multi INFO: found job 34092 in provider local 2024-03-13 08:46:44 proc_lsst.multi:146 [INFO] found job 17008896 in provider astro proc_lsst.multi INFO: found job 17008896 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 160 try 0 failed parsl.dataflow.dflow INFO: Task 160 marked for retry parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898307/600d84f7-6e60-4406-8464-e4a9809afada_characterizeImage_898307_2.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898307/600d84f7-6e60-4406-8464-e4a9809afada_characterizeImage_898307_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 160 has memoization hash 45bc3cff8c8d52a100a544bcbdac6f1e parsl.dataflow.memoization INFO: Task 160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bd8e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 160 try 1 launched on executor multi with executor id 249 parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898307/600d84f7-6e60-4406-8464-e4a9809afada_characterizeImage_898307_2.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898307/600d84f7-6e60-4406-8464-e4a9809afada_characterizeImage_898307_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 172 try 0 failed parsl.dataflow.dflow INFO: Task 172 marked for retry parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898291/3f08e496-3c1c-4e7b-81b8-e00bfd49915c_characterizeImage_898291_2.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898291/3f08e496-3c1c-4e7b-81b8-e00bfd49915c_characterizeImage_898291_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 172 has memoization hash 82a7632dea989fb59b9debf483e8f257 parsl.dataflow.memoization INFO: Task 172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bd9940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 172 try 1 launched on executor multi with executor id 250 parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898291/3f08e496-3c1c-4e7b-81b8-e00bfd49915c_characterizeImage_898291_2.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898291/3f08e496-3c1c-4e7b-81b8-e00bfd49915c_characterizeImage_898291_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 166 try 0 failed parsl.dataflow.dflow INFO: Task 166 marked for retry parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898300/b94fef20-f629-47af-93e6-c9f2a3a9d58e_characterizeImage_898300_2.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898300/b94fef20-f629-47af-93e6-c9f2a3a9d58e_characterizeImage_898300_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 166 has memoization hash 9e37b0df012723d5c38d6fd39629e374 parsl.dataflow.memoization INFO: Task 166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bd9580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 166 try 1 launched on executor multi with executor id 251 parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898300/b94fef20-f629-47af-93e6-c9f2a3a9d58e_characterizeImage_898300_2.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898300/b94fef20-f629-47af-93e6-c9f2a3a9d58e_characterizeImage_898300_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 174 try 0 failed parsl.dataflow.dflow INFO: Task 174 marked for retry parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898325/a1d83ff3-a248-4b6d-a09b-46c0ec2b1db9_characterizeImage_898325_2.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898325/a1d83ff3-a248-4b6d-a09b-46c0ec2b1db9_characterizeImage_898325_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 174 has memoization hash 53d95f458579079a7083b5d9467898cf parsl.dataflow.memoization INFO: Task 174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bd9b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 174 try 1 launched on executor multi with executor id 252 parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898325/a1d83ff3-a248-4b6d-a09b-46c0ec2b1db9_characterizeImage_898325_2.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898325/a1d83ff3-a248-4b6d-a09b-46c0ec2b1db9_characterizeImage_898325_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 158 try 0 failed parsl.dataflow.dflow INFO: Task 158 marked for retry parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898368/402adb10-54c3-4a78-9606-4fcca2726950_characterizeImage_898368_2.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898368/402adb10-54c3-4a78-9606-4fcca2726950_characterizeImage_898368_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 158 has memoization hash a24c7ea12e2c2d17472c2e285c904e3c parsl.dataflow.memoization INFO: Task 158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bd8a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 158 try 1 launched on executor multi with executor id 253 parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898368/402adb10-54c3-4a78-9606-4fcca2726950_characterizeImage_898368_2.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898368/402adb10-54c3-4a78-9606-4fcca2726950_characterizeImage_898368_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:46:46 proc_lsst.multi:146 [INFO] found job 17008898 in provider astro proc_lsst.multi INFO: found job 17008898 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 170 try 0 failed parsl.dataflow.dflow INFO: Task 170 marked for retry parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898381/def23eee-f468-417e-a13d-2c3590830f1a_characterizeImage_898381_2.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898381/def23eee-f468-417e-a13d-2c3590830f1a_characterizeImage_898381_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 170 has memoization hash bb3d28e661ccb7a9e2497ea69d198be1 parsl.dataflow.memoization INFO: Task 170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bd9760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 170 try 1 launched on executor multi with executor id 254 parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898381/def23eee-f468-417e-a13d-2c3590830f1a_characterizeImage_898381_2.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898381/def23eee-f468-417e-a13d-2c3590830f1a_characterizeImage_898381_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 176 try 0 failed parsl.dataflow.dflow INFO: Task 176 marked for retry parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898289/826cbbf6-3f98-46d6-8550-55ac105a4087_characterizeImage_898289_2.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898289/826cbbf6-3f98-46d6-8550-55ac105a4087_characterizeImage_898289_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 176 has memoization hash 789a3cf4fbdb6926020ca79be33c4a3b parsl.dataflow.memoization INFO: Task 176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bd9ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 176 try 1 launched on executor multi with executor id 255 parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898289/826cbbf6-3f98-46d6-8550-55ac105a4087_characterizeImage_898289_2.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898289/826cbbf6-3f98-46d6-8550-55ac105a4087_characterizeImage_898289_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 178 try 0 failed parsl.dataflow.dflow INFO: Task 178 marked for retry parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898316/859c2f84-b5d9-4766-8e01-03ce74cd180a_characterizeImage_898316_2.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898316/859c2f84-b5d9-4766-8e01-03ce74cd180a_characterizeImage_898316_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 178 has memoization hash 5e6ed418d182a60ebe3b6707af1f3fb7 parsl.dataflow.memoization INFO: Task 178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bd9d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 178 try 1 launched on executor multi with executor id 256 parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898316/859c2f84-b5d9-4766-8e01-03ce74cd180a_characterizeImage_898316_2.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898316/859c2f84-b5d9-4766-8e01-03ce74cd180a_characterizeImage_898316_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 180 try 0 failed parsl.dataflow.dflow INFO: Task 180 marked for retry parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898340/d4cf30dc-0e16-4a99-b41a-89a591835063_characterizeImage_898340_2.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898340/d4cf30dc-0e16-4a99-b41a-89a591835063_characterizeImage_898340_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 180 has memoization hash 83183bddadbe819dd868ad0180af8efb parsl.dataflow.memoization INFO: Task 180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bda0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 180 try 1 launched on executor multi with executor id 257 parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898340/d4cf30dc-0e16-4a99-b41a-89a591835063_characterizeImage_898340_2.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898340/d4cf30dc-0e16-4a99-b41a-89a591835063_characterizeImage_898340_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:46:47 proc_lsst.multi:146 [INFO] found job 17008899 in provider astro proc_lsst.multi INFO: found job 17008899 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 188 try 0 failed parsl.dataflow.dflow INFO: Task 188 marked for retry parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898304/f4fe1812-be36-41cd-8197-d661a0fcd8c9_characterizeImage_898304_2.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898304/f4fe1812-be36-41cd-8197-d661a0fcd8c9_characterizeImage_898304_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 188 has memoization hash 3732fc702b75d329c601489899e3f640 parsl.dataflow.memoization INFO: Task 188 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bda660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 188 try 1 launched on executor multi with executor id 258 parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898304/f4fe1812-be36-41cd-8197-d661a0fcd8c9_characterizeImage_898304_2.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898304/f4fe1812-be36-41cd-8197-d661a0fcd8c9_characterizeImage_898304_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 168 try 0 failed parsl.dataflow.dflow INFO: Task 168 marked for retry parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898353/8b3003ab-d982-40fb-9a29-d42d3753c93a_characterizeImage_898353_2.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898353/8b3003ab-d982-40fb-9a29-d42d3753c93a_characterizeImage_898353_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 168 has memoization hash b04f9d26fa7a7033635a9284cede03de parsl.dataflow.memoization INFO: Task 168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bd93a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 168 try 1 launched on executor multi with executor id 259 parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898353/8b3003ab-d982-40fb-9a29-d42d3753c93a_characterizeImage_898353_2.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898353/8b3003ab-d982-40fb-9a29-d42d3753c93a_characterizeImage_898353_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 194 try 0 failed parsl.dataflow.dflow INFO: Task 194 marked for retry parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898366/883e1fdb-0061-4803-a77b-f6a737d0eb4d_characterizeImage_898366_2.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898366/883e1fdb-0061-4803-a77b-f6a737d0eb4d_characterizeImage_898366_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 194 has memoization hash c862e51217a6299650d85a3e39d0d8d7 parsl.dataflow.memoization INFO: Task 194 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bdade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 194 try 1 launched on executor multi with executor id 260 parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898366/883e1fdb-0061-4803-a77b-f6a737d0eb4d_characterizeImage_898366_2.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898366/883e1fdb-0061-4803-a77b-f6a737d0eb4d_characterizeImage_898366_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 184 try 0 failed parsl.dataflow.dflow INFO: Task 184 marked for retry parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898374/7d868f31-aea1-4df3-bca4-b4f657a601a8_characterizeImage_898374_2.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898374/7d868f31-aea1-4df3-bca4-b4f657a601a8_characterizeImage_898374_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 184 has memoization hash c3b73f90ebf119d57db71ad1aae6ce4e parsl.dataflow.memoization INFO: Task 184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bda480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 184 try 1 launched on executor multi with executor id 261 parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898374/7d868f31-aea1-4df3-bca4-b4f657a601a8_characterizeImage_898374_2.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898374/7d868f31-aea1-4df3-bca4-b4f657a601a8_characterizeImage_898374_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 182 try 0 failed parsl.dataflow.dflow INFO: Task 182 marked for retry parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898320/0d271a51-6393-4e3c-8de9-ccd83369b17f_characterizeImage_898320_2.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898320/0d271a51-6393-4e3c-8de9-ccd83369b17f_characterizeImage_898320_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 182 has memoization hash 187009d2c954513b50496652950b3e61 parsl.dataflow.memoization INFO: Task 182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bda2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 182 try 1 launched on executor multi with executor id 262 parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898320/0d271a51-6393-4e3c-8de9-ccd83369b17f_characterizeImage_898320_2.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898320/0d271a51-6393-4e3c-8de9-ccd83369b17f_characterizeImage_898320_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 196 try 0 failed parsl.dataflow.dflow INFO: Task 196 marked for retry parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898313/8efe628c-fbcb-4620-b8a1-da84903ac46a_characterizeImage_898313_2.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898313/8efe628c-fbcb-4620-b8a1-da84903ac46a_characterizeImage_898313_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 196 has memoization hash 7c164b614258f56e5d1018602d2c6b22 parsl.dataflow.memoization INFO: Task 196 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bdb1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 196 try 1 launched on executor multi with executor id 263 parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898313/8efe628c-fbcb-4620-b8a1-da84903ac46a_characterizeImage_898313_2.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898313/8efe628c-fbcb-4620-b8a1-da84903ac46a_characterizeImage_898313_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 200 try 0 failed parsl.dataflow.dflow INFO: Task 200 marked for retry parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898347/cc7819d0-5467-422c-b477-88c64cf6b601_characterizeImage_898347_2.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898347/cc7819d0-5467-422c-b477-88c64cf6b601_characterizeImage_898347_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 200 has memoization hash 98a6b5e9eb625aa688a4be5b26f90646 parsl.dataflow.memoization INFO: Task 200 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bdb380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 200 try 1 launched on executor multi with executor id 264 parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898347/cc7819d0-5467-422c-b477-88c64cf6b601_characterizeImage_898347_2.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898347/cc7819d0-5467-422c-b477-88c64cf6b601_characterizeImage_898347_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 186 try 0 failed parsl.dataflow.dflow INFO: Task 186 marked for retry parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898331/c0c18cff-9b6d-4db0-9644-84341b2c3bbd_characterizeImage_898331_2.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898331/c0c18cff-9b6d-4db0-9644-84341b2c3bbd_characterizeImage_898331_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 186 has memoization hash e6bf795532176e2d27dad486567f1e64 parsl.dataflow.memoization INFO: Task 186 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bda840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 186 try 1 launched on executor multi with executor id 265 parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898331/c0c18cff-9b6d-4db0-9644-84341b2c3bbd_characterizeImage_898331_2.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898331/c0c18cff-9b6d-4db0-9644-84341b2c3bbd_characterizeImage_898331_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:46:49 proc_lsst.multi:146 [INFO] found job 17008900 in provider astro proc_lsst.multi INFO: found job 17008900 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 24 try 0 failed parsl.dataflow.dflow INFO: Task 24 marked for retry parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898331/5c785658-adcd-419d-b339-11f13f7ccddd_calibrate_898331_61.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898331/5c785658-adcd-419d-b339-11f13f7ccddd_calibrate_898331_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 24 has memoization hash 4a0f9e7707a025e86a9bce98ffc50f66 parsl.dataflow.memoization INFO: Task 24 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b69760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 24 try 1 launched on executor multi with executor id 266 parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898331/5c785658-adcd-419d-b339-11f13f7ccddd_calibrate_898331_61.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898331/5c785658-adcd-419d-b339-11f13f7ccddd_calibrate_898331_61.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 192 try 0 failed parsl.dataflow.dflow INFO: Task 192 marked for retry parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898338/cb25c482-36e9-4bee-a408-37e1611db7cf_characterizeImage_898338_2.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898338/cb25c482-36e9-4bee-a408-37e1611db7cf_characterizeImage_898338_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 192 has memoization hash 20726f9423ee27a4b04345ffe73518b7 parsl.dataflow.memoization INFO: Task 192 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bdac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 192 try 1 launched on executor multi with executor id 267 parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898338/cb25c482-36e9-4bee-a408-37e1611db7cf_characterizeImage_898338_2.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898338/cb25c482-36e9-4bee-a408-37e1611db7cf_characterizeImage_898338_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 190 try 0 failed parsl.dataflow.dflow INFO: Task 190 marked for retry parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898341/1ff0f251-6bb7-4969-aad8-00b357877c1c_characterizeImage_898341_2.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898341/1ff0f251-6bb7-4969-aad8-00b357877c1c_characterizeImage_898341_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 190 has memoization hash 3492ba1a18c1958aa8dd61434111c302 parsl.dataflow.memoization INFO: Task 190 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bdaa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 190 try 1 launched on executor multi with executor id 268 parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898341/1ff0f251-6bb7-4969-aad8-00b357877c1c_characterizeImage_898341_2.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898341/1ff0f251-6bb7-4969-aad8-00b357877c1c_characterizeImage_898341_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 198 try 0 failed parsl.dataflow.dflow INFO: Task 198 marked for retry parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898382/fdd9caee-a9d8-47b5-8d09-8b6e064291a9_characterizeImage_898382_2.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898382/fdd9caee-a9d8-47b5-8d09-8b6e064291a9_characterizeImage_898382_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 198 has memoization hash 9860e33d4639157e87123b78bcea19b9 parsl.dataflow.memoization INFO: Task 198 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bdafc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 198 try 1 launched on executor multi with executor id 269 parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898382/fdd9caee-a9d8-47b5-8d09-8b6e064291a9_characterizeImage_898382_2.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898382/fdd9caee-a9d8-47b5-8d09-8b6e064291a9_characterizeImage_898382_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 45 try 0 failed parsl.dataflow.dflow INFO: Task 45 marked for retry parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898322/917bf878-02ae-462d-8477-3109632be425_calibrate_898322_61.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898322/917bf878-02ae-462d-8477-3109632be425_calibrate_898322_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 45 has memoization hash 98fd1d3c7031e1dbd8391f88c1a617db parsl.dataflow.memoization INFO: Task 45 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 45 try 1 launched on executor multi with executor id 270 parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898322/917bf878-02ae-462d-8477-3109632be425_calibrate_898322_61.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898322/917bf878-02ae-462d-8477-3109632be425_calibrate_898322_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:46:50 proc_lsst.multi:146 [INFO] found job 17008901 in provider astro proc_lsst.multi INFO: found job 17008901 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 46 try 0 failed parsl.dataflow.dflow INFO: Task 46 marked for retry parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898332/1afe740a-c14b-4016-ba53-e7fbd2c9f2fc_calibrate_898332_61.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898332/1afe740a-c14b-4016-ba53-e7fbd2c9f2fc_calibrate_898332_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 46 has memoization hash 8f8e36e3614cf9d2377fccbbc80d45b4 parsl.dataflow.memoization INFO: Task 46 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 46 try 1 launched on executor multi with executor id 271 parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898332/1afe740a-c14b-4016-ba53-e7fbd2c9f2fc_calibrate_898332_61.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898332/1afe740a-c14b-4016-ba53-e7fbd2c9f2fc_calibrate_898332_61.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 208 try 0 failed parsl.dataflow.dflow INFO: Task 208 marked for retry parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898378/0995fe10-ea29-4d67-a721-fb8b6cd22e66_characterizeImage_898378_2.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898378/0995fe10-ea29-4d67-a721-fb8b6cd22e66_characterizeImage_898378_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 208 has memoization hash 87aac7584d5218dd9b19f278ccc6e0d5 parsl.dataflow.memoization INFO: Task 208 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bdb920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 208 try 1 launched on executor multi with executor id 272 parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898378/0995fe10-ea29-4d67-a721-fb8b6cd22e66_characterizeImage_898378_2.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898378/0995fe10-ea29-4d67-a721-fb8b6cd22e66_characterizeImage_898378_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 206 try 0 failed parsl.dataflow.dflow INFO: Task 206 marked for retry parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898292/3ca39bab-f6c2-4bd5-9b58-06747c252ce8_characterizeImage_898292_2.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898292/3ca39bab-f6c2-4bd5-9b58-06747c252ce8_characterizeImage_898292_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 206 has memoization hash 47cbca9dbf88bdcdc02765303207aee9 parsl.dataflow.memoization INFO: Task 206 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bdbb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 206 try 1 launched on executor multi with executor id 273 parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898292/3ca39bab-f6c2-4bd5-9b58-06747c252ce8_characterizeImage_898292_2.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898292/3ca39bab-f6c2-4bd5-9b58-06747c252ce8_characterizeImage_898292_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 210 try 0 failed parsl.dataflow.dflow INFO: Task 210 marked for retry parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898379/64202fb7-6336-4e9f-8200-02d1651dce8f_characterizeImage_898379_2.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898379/64202fb7-6336-4e9f-8200-02d1651dce8f_characterizeImage_898379_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 210 has memoization hash 37c00a91549aa1477fe31946f97a7c5f parsl.dataflow.memoization INFO: Task 210 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bdbce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 210 try 1 launched on executor multi with executor id 274 parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898379/64202fb7-6336-4e9f-8200-02d1651dce8f_characterizeImage_898379_2.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898379/64202fb7-6336-4e9f-8200-02d1651dce8f_characterizeImage_898379_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 202 try 0 failed parsl.dataflow.dflow INFO: Task 202 marked for retry parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898305/6eb2b7de-5132-4dab-852e-cf7330151622_characterizeImage_898305_2.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898305/6eb2b7de-5132-4dab-852e-cf7330151622_characterizeImage_898305_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 202 has memoization hash 5856865b095f60519def504b9848d2e8 parsl.dataflow.memoization INFO: Task 202 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bdb560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 202 try 1 launched on executor multi with executor id 275 parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898305/6eb2b7de-5132-4dab-852e-cf7330151622_characterizeImage_898305_2.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898305/6eb2b7de-5132-4dab-852e-cf7330151622_characterizeImage_898305_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 214 try 0 failed parsl.dataflow.dflow INFO: Task 214 marked for retry parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898351/8862f32c-0af5-4ab1-84b5-911135b20c05_characterizeImage_898351_2.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898351/8862f32c-0af5-4ab1-84b5-911135b20c05_characterizeImage_898351_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 214 has memoization hash 9e891114e331dc0777c030114d61ec77 parsl.dataflow.memoization INFO: Task 214 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394c0c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 214 try 1 launched on executor multi with executor id 276 parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898351/8862f32c-0af5-4ab1-84b5-911135b20c05_characterizeImage_898351_2.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898351/8862f32c-0af5-4ab1-84b5-911135b20c05_characterizeImage_898351_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 204 try 0 failed parsl.dataflow.dflow INFO: Task 204 marked for retry parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898308/243ba183-700c-4b37-9a39-4b8717e68bde_characterizeImage_898308_2.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898308/243ba183-700c-4b37-9a39-4b8717e68bde_characterizeImage_898308_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 204 has memoization hash 24d2c9716569392e6d623f504981bd3b parsl.dataflow.memoization INFO: Task 204 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394bdb740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 204 try 1 launched on executor multi with executor id 277 parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898308/243ba183-700c-4b37-9a39-4b8717e68bde_characterizeImage_898308_2.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898308/243ba183-700c-4b37-9a39-4b8717e68bde_characterizeImage_898308_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:46:51 proc_lsst.multi:146 [INFO] found job 17008902 in provider astro proc_lsst.multi INFO: found job 17008902 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 218 try 0 failed parsl.dataflow.dflow INFO: Task 218 marked for retry parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898372/92791b56-4fa4-4dbb-928e-3fbe354d21b6_characterizeImage_898372_2.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898372/92791b56-4fa4-4dbb-928e-3fbe354d21b6_characterizeImage_898372_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 218 has memoization hash 7cf33828fc86323e1122578488096e86 parsl.dataflow.memoization INFO: Task 218 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394c0c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 218 try 1 launched on executor multi with executor id 278 parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898372/92791b56-4fa4-4dbb-928e-3fbe354d21b6_characterizeImage_898372_2.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898372/92791b56-4fa4-4dbb-928e-3fbe354d21b6_characterizeImage_898372_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 212 try 0 failed parsl.dataflow.dflow INFO: Task 212 marked for retry parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898336/59045ea5-e278-4d7d-a033-56463a2f5fac_characterizeImage_898336_2.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898336/59045ea5-e278-4d7d-a033-56463a2f5fac_characterizeImage_898336_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 212 has memoization hash 5dcdaaa9962234e2d4da66b378702105 parsl.dataflow.memoization INFO: Task 212 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394c0c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 212 try 1 launched on executor multi with executor id 279 parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898336/59045ea5-e278-4d7d-a033-56463a2f5fac_characterizeImage_898336_2.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898336/59045ea5-e278-4d7d-a033-56463a2f5fac_characterizeImage_898336_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 216 try 0 failed parsl.dataflow.dflow INFO: Task 216 marked for retry parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898363/15ceb904-a4e7-4b48-ac94-ffecf8eeed6b_characterizeImage_898363_2.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898363/15ceb904-a4e7-4b48-ac94-ffecf8eeed6b_characterizeImage_898363_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 216 has memoization hash f714afe43fbdf6119189762e695b5bfe parsl.dataflow.memoization INFO: Task 216 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394c0c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 216 try 1 launched on executor multi with executor id 280 parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898363/15ceb904-a4e7-4b48-ac94-ffecf8eeed6b_characterizeImage_898363_2.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898363/15ceb904-a4e7-4b48-ac94-ffecf8eeed6b_characterizeImage_898363_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:46:53 proc_lsst.multi:146 [INFO] found job 17008903 in provider astro proc_lsst.multi INFO: found job 17008903 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 230 try 0 failed parsl.dataflow.dflow INFO: Task 230 marked for retry parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898367/1d9ca05f-cb75-4481-9d04-7f4a93801fb7_characterizeImage_898367_2.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898367/1d9ca05f-cb75-4481-9d04-7f4a93801fb7_characterizeImage_898367_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 230 has memoization hash c7b4b8af22ecad1d67932da5957e2f6c parsl.dataflow.memoization INFO: Task 230 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394c0d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 230 try 1 launched on executor multi with executor id 281 parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898367/1d9ca05f-cb75-4481-9d04-7f4a93801fb7_characterizeImage_898367_2.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898367/1d9ca05f-cb75-4481-9d04-7f4a93801fb7_characterizeImage_898367_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 222 try 0 failed parsl.dataflow.dflow INFO: Task 222 marked for retry parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898299/a64d3126-e727-422d-a274-393d2146b227_characterizeImage_898299_2.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898299/a64d3126-e727-422d-a274-393d2146b227_characterizeImage_898299_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 222 has memoization hash b6a5488867a8c941e81fba6ca799cbd6 parsl.dataflow.memoization INFO: Task 222 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394c0c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 222 try 1 launched on executor multi with executor id 282 parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898299/a64d3126-e727-422d-a274-393d2146b227_characterizeImage_898299_2.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898299/a64d3126-e727-422d-a274-393d2146b227_characterizeImage_898299_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 51 try 0 failed parsl.dataflow.dflow INFO: Task 51 marked for retry parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898353/b15df392-ec28-447e-81c3-72d987d085c6_calibrate_898353_61.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898353/b15df392-ec28-447e-81c3-72d987d085c6_calibrate_898353_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 51 has memoization hash 0a6b63a644cafb3f45916f28e2dcab00 parsl.dataflow.memoization INFO: Task 51 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 51 try 1 launched on executor multi with executor id 283 parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898353/b15df392-ec28-447e-81c3-72d987d085c6_calibrate_898353_61.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898353/b15df392-ec28-447e-81c3-72d987d085c6_calibrate_898353_61.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 226 try 0 failed parsl.dataflow.dflow INFO: Task 226 marked for retry parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898329/9a25d39c-9128-4551-b478-cd5d38257f0d_characterizeImage_898329_2.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898329/9a25d39c-9128-4551-b478-cd5d38257f0d_characterizeImage_898329_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 226 has memoization hash 70e29d359f3a79d340340ba3825a7d7c parsl.dataflow.memoization INFO: Task 226 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394c0ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 226 try 1 launched on executor multi with executor id 284 parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898329/9a25d39c-9128-4551-b478-cd5d38257f0d_characterizeImage_898329_2.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898329/9a25d39c-9128-4551-b478-cd5d38257f0d_characterizeImage_898329_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 240 try 0 failed parsl.dataflow.dflow INFO: Task 240 marked for retry parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898361/70cb0c22-8082-4558-a092-2d2526583506_characterizeImage_898361_2.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898361/70cb0c22-8082-4558-a092-2d2526583506_characterizeImage_898361_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 240 has memoization hash 0ce1d3fee239f4d14df437cdacf4f103 parsl.dataflow.memoization INFO: Task 240 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394c0dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 240 try 1 launched on executor multi with executor id 285 parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898361/70cb0c22-8082-4558-a092-2d2526583506_characterizeImage_898361_2.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898361/70cb0c22-8082-4558-a092-2d2526583506_characterizeImage_898361_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 224 try 0 failed parsl.dataflow.dflow INFO: Task 224 marked for retry parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898359/65636a59-80fb-47c1-8f98-43c4f999b198_characterizeImage_898359_2.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898359/65636a59-80fb-47c1-8f98-43c4f999b198_characterizeImage_898359_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 224 has memoization hash 18f9bb934b3848c102c6446fb94941ad parsl.dataflow.memoization INFO: Task 224 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394c0ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 224 try 1 launched on executor multi with executor id 286 parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898359/65636a59-80fb-47c1-8f98-43c4f999b198_characterizeImage_898359_2.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898359/65636a59-80fb-47c1-8f98-43c4f999b198_characterizeImage_898359_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 234 try 0 failed parsl.dataflow.dflow INFO: Task 234 marked for retry parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898309/ab2a58da-d00a-4358-b4f7-75fcdce6e2e5_characterizeImage_898309_2.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898309/ab2a58da-d00a-4358-b4f7-75fcdce6e2e5_characterizeImage_898309_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 234 has memoization hash 9847deae22a0da9db67da6a661dd1026 parsl.dataflow.memoization INFO: Task 234 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394c0d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:46:54 proc_lsst.multi:146 [INFO] found job 17008904 in provider astro proc_lsst.multi INFO: found job 17008904 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.dataflow.dflow INFO: Parsl task 234 try 1 launched on executor multi with executor id 287 parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898309/ab2a58da-d00a-4358-b4f7-75fcdce6e2e5_characterizeImage_898309_2.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898309/ab2a58da-d00a-4358-b4f7-75fcdce6e2e5_characterizeImage_898309_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 53 try 0 failed parsl.dataflow.dflow INFO: Task 53 marked for retry parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898346/148e6dc2-a926-488e-8cfb-2a923573e179_calibrate_898346_61.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898346/148e6dc2-a926-488e-8cfb-2a923573e179_calibrate_898346_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 53 has memoization hash 5643377b5f14ba72e85f5850cb6406a2 parsl.dataflow.memoization INFO: Task 53 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 53 try 1 launched on executor multi with executor id 288 parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898346/148e6dc2-a926-488e-8cfb-2a923573e179_calibrate_898346_61.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898346/148e6dc2-a926-488e-8cfb-2a923573e179_calibrate_898346_61.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 228 try 0 failed parsl.dataflow.dflow INFO: Task 228 marked for retry parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898343/af00f831-886a-4c91-aed5-e90dc3c54710_characterizeImage_898343_2.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898343/af00f831-886a-4c91-aed5-e90dc3c54710_characterizeImage_898343_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 228 has memoization hash 5c901b5f51cce774e36848a802b4353c parsl.dataflow.memoization INFO: Task 228 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394c0cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 228 try 1 launched on executor multi with executor id 289 parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898343/af00f831-886a-4c91-aed5-e90dc3c54710_characterizeImage_898343_2.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898343/af00f831-886a-4c91-aed5-e90dc3c54710_characterizeImage_898343_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 242 try 0 failed parsl.dataflow.dflow INFO: Task 242 marked for retry parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898314/0c096c3b-1ae9-4535-9515-ed489ad83844_characterizeImage_898314_2.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898314/0c096c3b-1ae9-4535-9515-ed489ad83844_characterizeImage_898314_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 242 has memoization hash 4074b8db1a55d97a7306e221d12c8572 parsl.dataflow.memoization INFO: Task 242 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394c0dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 242 try 1 launched on executor multi with executor id 290 parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898314/0c096c3b-1ae9-4535-9515-ed489ad83844_characterizeImage_898314_2.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898314/0c096c3b-1ae9-4535-9515-ed489ad83844_characterizeImage_898314_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 220 try 0 failed parsl.dataflow.dflow INFO: Task 220 marked for retry parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898335/281503df-63c1-4918-888e-a90e0e4d90ea_characterizeImage_898335_2.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898335/281503df-63c1-4918-888e-a90e0e4d90ea_characterizeImage_898335_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 220 has memoization hash a2683b89b666f946be5df544b264d391 parsl.dataflow.memoization INFO: Task 220 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394c0c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 220 try 1 launched on executor multi with executor id 291 parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898335/281503df-63c1-4918-888e-a90e0e4d90ea_characterizeImage_898335_2.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898335/281503df-63c1-4918-888e-a90e0e4d90ea_characterizeImage_898335_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 236 try 0 failed parsl.dataflow.dflow INFO: Task 236 marked for retry parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898298/a0281b0b-1e39-406c-851a-d3046c548e4c_characterizeImage_898298_2.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898298/a0281b0b-1e39-406c-851a-d3046c548e4c_characterizeImage_898298_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 236 has memoization hash 6f9c150e9df8027239798bcf46aa33f0 parsl.dataflow.memoization INFO: Task 236 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394c0db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 236 try 1 launched on executor multi with executor id 292 parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898298/a0281b0b-1e39-406c-851a-d3046c548e4c_characterizeImage_898298_2.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898298/a0281b0b-1e39-406c-851a-d3046c548e4c_characterizeImage_898298_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 238 try 0 failed parsl.dataflow.dflow INFO: Task 238 marked for retry parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898296/a19de25d-9b4d-4608-b993-e76d6f64d771_characterizeImage_898296_2.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898296/a19de25d-9b4d-4608-b993-e76d6f64d771_characterizeImage_898296_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 238 has memoization hash 23c3f8d39c7bb146f77fc858d2f88f77 parsl.dataflow.memoization INFO: Task 238 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394c0d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 238 try 1 launched on executor multi with executor id 293 parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898296/a19de25d-9b4d-4608-b993-e76d6f64d771_characterizeImage_898296_2.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898296/a19de25d-9b4d-4608-b993-e76d6f64d771_characterizeImage_898296_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 244 try 0 failed parsl.dataflow.dflow INFO: Task 244 marked for retry parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898377/73990b84-2916-4ebb-b290-1fbd20e627f9_characterizeImage_898377_2.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898377/73990b84-2916-4ebb-b290-1fbd20e627f9_characterizeImage_898377_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 244 has memoization hash 157cf90d46774aee3d879c243e268d49 parsl.dataflow.memoization INFO: Task 244 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394c0e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 244 try 1 launched on executor multi with executor id 294 parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898377/73990b84-2916-4ebb-b290-1fbd20e627f9_characterizeImage_898377_2.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898377/73990b84-2916-4ebb-b290-1fbd20e627f9_characterizeImage_898377_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:46:56 proc_lsst.multi:146 [INFO] found job 17008905 in provider astro proc_lsst.multi INFO: found job 17008905 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 48 try 0 failed parsl.dataflow.dflow INFO: Task 48 marked for retry parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898382/26d7638c-432a-48eb-8a5e-2bd420831861_calibrate_898382_61.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898382/26d7638c-432a-48eb-8a5e-2bd420831861_calibrate_898382_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 48 has memoization hash 81651c4a5cc7c31adf702f2a62e10b61 parsl.dataflow.memoization INFO: Task 48 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 48 try 1 launched on executor multi with executor id 295 parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898382/26d7638c-432a-48eb-8a5e-2bd420831861_calibrate_898382_61.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898382/26d7638c-432a-48eb-8a5e-2bd420831861_calibrate_898382_61.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 47 try 0 failed parsl.dataflow.dflow INFO: Task 47 marked for retry parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898365/68f3688a-de27-4713-83e4-4a8332c2e510_calibrate_898365_61.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898365/68f3688a-de27-4713-83e4-4a8332c2e510_calibrate_898365_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 47 has memoization hash 495796d6dd275cdebea5a7947f2db62b parsl.dataflow.memoization INFO: Task 47 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6afc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 47 try 1 launched on executor multi with executor id 296 parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898365/68f3688a-de27-4713-83e4-4a8332c2e510_calibrate_898365_61.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898365/68f3688a-de27-4713-83e4-4a8332c2e510_calibrate_898365_61.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 246 try 0 failed parsl.dataflow.dflow INFO: Task 246 marked for retry parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898357/5a7bd7a2-a9a9-41a8-9bb6-6917a13f6a9b_characterizeImage_898357_2.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898357/5a7bd7a2-a9a9-41a8-9bb6-6917a13f6a9b_characterizeImage_898357_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 246 has memoization hash 210aca5b3c6c55476d15778b06af4561 parsl.dataflow.memoization INFO: Task 246 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394c0e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 246 try 1 launched on executor multi with executor id 297 parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898357/5a7bd7a2-a9a9-41a8-9bb6-6917a13f6a9b_characterizeImage_898357_2.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898357/5a7bd7a2-a9a9-41a8-9bb6-6917a13f6a9b_characterizeImage_898357_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 248 try 0 failed parsl.dataflow.dflow INFO: Task 248 marked for retry parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898364/54421108-f5fa-4a91-9cb8-668488e2f15d_characterizeImage_898364_2.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898364/54421108-f5fa-4a91-9cb8-668488e2f15d_characterizeImage_898364_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 248 has memoization hash 66068bc7a08514af0d0332311219ddb2 parsl.dataflow.memoization INFO: Task 248 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394c0d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 248 try 1 launched on executor multi with executor id 298 parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898364/54421108-f5fa-4a91-9cb8-668488e2f15d_characterizeImage_898364_2.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898364/54421108-f5fa-4a91-9cb8-668488e2f15d_characterizeImage_898364_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 41 try 0 failed parsl.dataflow.dflow INFO: Task 41 marked for retry parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898324/8d56e6fa-7a2c-4b9b-afa3-e24d76c7887b_calibrate_898324_61.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898324/8d56e6fa-7a2c-4b9b-afa3-e24d76c7887b_calibrate_898324_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 41 has memoization hash 9ca2ae210d2c757b4188bbf1deab3a05 parsl.dataflow.memoization INFO: Task 41 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 41 try 1 launched on executor multi with executor id 299 parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898324/8d56e6fa-7a2c-4b9b-afa3-e24d76c7887b_calibrate_898324_61.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898324/8d56e6fa-7a2c-4b9b-afa3-e24d76c7887b_calibrate_898324_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:46:57 proc_lsst.multi:146 [INFO] found job 17008906 in provider astro proc_lsst.multi INFO: found job 17008906 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 232 try 0 failed parsl.dataflow.dflow INFO: Task 232 marked for retry parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898348/8791bf45-0123-47d0-97f9-3f528018f72b_characterizeImage_898348_2.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898348/8791bf45-0123-47d0-97f9-3f528018f72b_characterizeImage_898348_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 232 has memoization hash 88c19e98bc647589cf501c6eef68b0f9 parsl.dataflow.memoization INFO: Task 232 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394c0d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 232 try 1 launched on executor multi with executor id 300 parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898348/8791bf45-0123-47d0-97f9-3f528018f72b_characterizeImage_898348_2.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898348/8791bf45-0123-47d0-97f9-3f528018f72b_characterizeImage_898348_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:46:58 proc_lsst.multi:146 [INFO] found job 17008907 in provider astro proc_lsst.multi INFO: found job 17008907 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:47:00 proc_lsst.multi:146 [INFO] found job 17008908 in provider astro proc_lsst.multi INFO: found job 17008908 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 56 try 1 failed parsl.dataflow.dflow ERROR: Task 56 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 8a5856ee7d0c61a4546900cabfbfe1c9 with result from task 56 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 57 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 57 try 0 failed parsl.dataflow.dflow INFO: Task 57 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898301/f1deaa36-ee0b-4ebb-90f7-9c9a19934dda_calibrate_898301_2.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898301/f1deaa36-ee0b-4ebb-90f7-9c9a19934dda_calibrate_898301_2.stderr parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898301/8ac197d9-bd72-461c-90c7-d994ec4e890c_characterizeImage_898301_2.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898301/8ac197d9-bd72-461c-90c7-d994ec4e890c_characterizeImage_898301_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:47:01 proc_lsst.multi:146 [INFO] found job 17008909 in provider astro proc_lsst.multi INFO: found job 17008909 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:47:02 proc_lsst.multi:146 [INFO] found job 17008910 in provider astro proc_lsst.multi INFO: found job 17008910 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 58 try 1 failed parsl.dataflow.dflow ERROR: Task 58 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 871f1c0bfc2b03edddb157476d0651fe with result from task 58 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 59 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 59 try 0 failed parsl.dataflow.dflow INFO: Task 59 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898311/691b230f-10ac-46ae-a8ba-796358325f2e_calibrate_898311_2.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898311/691b230f-10ac-46ae-a8ba-796358325f2e_calibrate_898311_2.stderr parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898311/72116209-a5e5-4f23-a885-45dbd5e314e6_characterizeImage_898311_2.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898311/72116209-a5e5-4f23-a885-45dbd5e314e6_characterizeImage_898311_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 62 try 1 failed parsl.dataflow.dflow ERROR: Task 62 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 2aba5b874159f2b9bd7bb6adc9fa55da with result from task 62 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 63 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 63 try 0 failed parsl.dataflow.dflow INFO: Task 63 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898306/1727ca38-2ef1-4189-9cb2-3dfd13b9aa73_calibrate_898306_2.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898306/1727ca38-2ef1-4189-9cb2-3dfd13b9aa73_calibrate_898306_2.stderr parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898306/687414c1-d1e9-4f0b-9b67-b80b325c7bb7_characterizeImage_898306_2.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898306/687414c1-d1e9-4f0b-9b67-b80b325c7bb7_characterizeImage_898306_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:47:04 proc_lsst.multi:146 [INFO] found job 17008911 in provider astro proc_lsst.multi INFO: found job 17008911 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 37 try 0 failed parsl.dataflow.dflow INFO: Task 37 marked for retry parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898329/0ba2197b-c2e4-4473-a0f4-3c1dedb841c1_calibrate_898329_61.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898329/0ba2197b-c2e4-4473-a0f4-3c1dedb841c1_calibrate_898329_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 37 has memoization hash 9bbeccbae9becdd64a681642865bf01b parsl.dataflow.memoization INFO: Task 37 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 37 try 1 launched on executor multi with executor id 301 parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898329/0ba2197b-c2e4-4473-a0f4-3c1dedb841c1_calibrate_898329_61.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898329/0ba2197b-c2e4-4473-a0f4-3c1dedb841c1_calibrate_898329_61.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 39 try 0 failed parsl.dataflow.dflow INFO: Task 39 marked for retry parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898341/22b89990-f444-407a-a373-03704d655796_calibrate_898341_61.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898341/22b89990-f444-407a-a373-03704d655796_calibrate_898341_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 39 has memoization hash 5d36b7e58d4a9ec85cab8d9824bd8fee parsl.dataflow.memoization INFO: Task 39 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 39 try 1 launched on executor multi with executor id 302 parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898341/22b89990-f444-407a-a373-03704d655796_calibrate_898341_61.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898341/22b89990-f444-407a-a373-03704d655796_calibrate_898341_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:47:05 proc_lsst.multi:146 [INFO] found job 17008913 in provider astro proc_lsst.multi INFO: found job 17008913 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:47:06 proc_lsst.multi:146 [INFO] found job 17008914 in provider astro proc_lsst.multi INFO: found job 17008914 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 70 try 1 failed parsl.dataflow.dflow ERROR: Task 70 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 3b2ca9a94f256047962755948959ef06 with result from task 70 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 71 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 71 try 0 failed parsl.dataflow.dflow INFO: Task 71 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898288/4adfb5e6-6d70-410f-af21-ee588f53e27b_calibrate_898288_2.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898288/4adfb5e6-6d70-410f-af21-ee588f53e27b_calibrate_898288_2.stderr parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898288/256364ee-b666-43fa-9aae-b3d74aa5385a_characterizeImage_898288_2.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898288/256364ee-b666-43fa-9aae-b3d74aa5385a_characterizeImage_898288_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 72 try 1 failed parsl.dataflow.dflow ERROR: Task 72 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 5e1fcaaa00a419d8d4e76a5c2bd888c7 with result from task 72 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 73 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 73 try 0 failed parsl.dataflow.dflow INFO: Task 73 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898375/569ee84b-3c18-46ec-9735-023218cf1e5b_calibrate_898375_2.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898375/569ee84b-3c18-46ec-9735-023218cf1e5b_calibrate_898375_2.stderr parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898375/9bad3368-9d28-420c-bd65-b5390f0dd3b5_characterizeImage_898375_2.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898375/9bad3368-9d28-420c-bd65-b5390f0dd3b5_characterizeImage_898375_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 50 try 0 failed parsl.dataflow.dflow INFO: Task 50 marked for retry parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898323/93fbc6ff-c819-4d23-b892-475cc0988494_calibrate_898323_61.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898323/93fbc6ff-c819-4d23-b892-475cc0988494_calibrate_898323_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 50 has memoization hash 362734b5a790c00cc0b62964f63e77c9 parsl.dataflow.memoization INFO: Task 50 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 50 try 1 launched on executor multi with executor id 303 parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898323/93fbc6ff-c819-4d23-b892-475cc0988494_calibrate_898323_61.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898323/93fbc6ff-c819-4d23-b892-475cc0988494_calibrate_898323_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:47:08 proc_lsst.multi:146 [INFO] found job 17008915 in provider astro proc_lsst.multi INFO: found job 17008915 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:47:09 proc_lsst.multi:146 [INFO] found job 17008917 in provider astro proc_lsst.multi INFO: found job 17008917 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 68 try 1 failed parsl.dataflow.dflow ERROR: Task 68 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 25e7d311cd1cde0309d3624f17af3185 with result from task 68 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 69 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 69 try 0 failed parsl.dataflow.dflow INFO: Task 69 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898334/d33b5123-3e26-4ed5-ad30-b89bd929add4_calibrate_898334_2.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898334/d33b5123-3e26-4ed5-ad30-b89bd929add4_calibrate_898334_2.stderr parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898334/cfea5b02-a172-4f43-a059-3065b6a16fe7_characterizeImage_898334_2.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898334/cfea5b02-a172-4f43-a059-3065b6a16fe7_characterizeImage_898334_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:47:10 proc_lsst.multi:146 [INFO] found job 17008918 in provider astro proc_lsst.multi INFO: found job 17008918 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 74 try 1 failed parsl.dataflow.dflow ERROR: Task 74 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 411d81f3ab7f15dad4bf228ec2a2a01a with result from task 74 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 75 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 75 try 0 failed parsl.dataflow.dflow INFO: Task 75 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898326/2022463c-07c0-4a58-a8af-8bf57f95e722_calibrate_898326_2.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898326/2022463c-07c0-4a58-a8af-8bf57f95e722_calibrate_898326_2.stderr parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898326/7d3152b1-6692-4435-9366-c053e4873078_characterizeImage_898326_2.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898326/7d3152b1-6692-4435-9366-c053e4873078_characterizeImage_898326_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:47:12 proc_lsst.multi:146 [INFO] found job 17008919 in provider astro proc_lsst.multi INFO: found job 17008919 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 64 try 1 failed parsl.dataflow.dflow ERROR: Task 64 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry cb65e91f672f3957fd9fc5594b209a2b with result from task 64 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 65 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 65 try 0 failed parsl.dataflow.dflow INFO: Task 65 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898380/f06c9554-3800-400b-b8c9-287cbb71904f_calibrate_898380_2.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898380/f06c9554-3800-400b-b8c9-287cbb71904f_calibrate_898380_2.stderr parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898380/90132901-dc69-42a9-86b3-bcd34dd462c2_characterizeImage_898380_2.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898380/90132901-dc69-42a9-86b3-bcd34dd462c2_characterizeImage_898380_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 60 try 1 failed parsl.dataflow.dflow ERROR: Task 60 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 97a9946e5ae719d5aaff1e8b4d89c2e8 with result from task 60 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 61 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 61 try 0 failed parsl.dataflow.dflow INFO: Task 61 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898290/44cd15f3-6944-4974-b8b8-e5f3d70b0c57_calibrate_898290_2.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898290/44cd15f3-6944-4974-b8b8-e5f3d70b0c57_calibrate_898290_2.stderr parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898290/ec58caa0-fe40-46e5-a79d-79858acbf8bc_characterizeImage_898290_2.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898290/ec58caa0-fe40-46e5-a79d-79858acbf8bc_characterizeImage_898290_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 78 try 1 failed parsl.dataflow.dflow ERROR: Task 78 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 9352761a2d2f84142e4de4fafc6a6e0e with result from task 78 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 79 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 79 try 0 failed parsl.dataflow.dflow INFO: Task 79 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898349/84acd277-4d3e-4661-a27c-140ab1ea1c04_calibrate_898349_2.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898349/84acd277-4d3e-4661-a27c-140ab1ea1c04_calibrate_898349_2.stderr parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898349/38416ffd-f62d-4e69-b703-f02832de1aea_characterizeImage_898349_2.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898349/38416ffd-f62d-4e69-b703-f02832de1aea_characterizeImage_898349_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:47:13 proc_lsst.multi:146 [INFO] found job 17008921 in provider astro proc_lsst.multi INFO: found job 17008921 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:47:14 proc_lsst.multi:146 [INFO] found job 17008922 in provider astro proc_lsst.multi INFO: found job 17008922 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 38 try 0 failed parsl.dataflow.dflow INFO: Task 38 marked for retry parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898337/44b1f232-8e71-4f34-8980-bb9b465c63b1_calibrate_898337_61.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898337/44b1f232-8e71-4f34-8980-bb9b465c63b1_calibrate_898337_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 38 has memoization hash 2cd84899ae596971a5725f38d92779eb parsl.dataflow.memoization INFO: Task 38 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 38 try 1 launched on executor multi with executor id 304 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898337/44b1f232-8e71-4f34-8980-bb9b465c63b1_calibrate_898337_61.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898337/44b1f232-8e71-4f34-8980-bb9b465c63b1_calibrate_898337_61.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 29 try 1 failed parsl.dataflow.dflow ERROR: Task 29 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 4039dedca7abdad7bd0d0159c6a786f7 with result from task 29 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898320/4d762e6c-fe69-40c5-a31f-25c11aca98cb_calibrate_898320_31.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898320/4d762e6c-fe69-40c5-a31f-25c11aca98cb_calibrate_898320_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:47:16 proc_lsst.multi:146 [INFO] found job 17008923 in provider astro proc_lsst.multi INFO: found job 17008923 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 3 try 1 failed parsl.dataflow.dflow ERROR: Task 3 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry ac45f7da037ff0da36555fa281bb6ff8 with result from task 3 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898350/ac90d1dd-77a7-4e35-9414-e727205aec56_calibrate_898350_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898350/ac90d1dd-77a7-4e35-9414-e727205aec56_calibrate_898350_31.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 66 try 1 failed parsl.dataflow.dflow ERROR: Task 66 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 1c3e7bb0f533db41b0d33e340d0bfcc4 with result from task 66 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 67 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 67 try 0 failed parsl.dataflow.dflow INFO: Task 67 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898355/68bb1765-13e3-4f67-9eb1-df2a60fcdebf_calibrate_898355_2.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898355/68bb1765-13e3-4f67-9eb1-df2a60fcdebf_calibrate_898355_2.stderr parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898355/10771c4c-8a1e-4e1e-9245-87f11f5ccd3e_characterizeImage_898355_2.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898355/10771c4c-8a1e-4e1e-9245-87f11f5ccd3e_characterizeImage_898355_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 76 try 1 failed parsl.dataflow.dflow ERROR: Task 76 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 3afd370b25175e2b9b252e3249fab9ea with result from task 76 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 77 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 77 try 0 failed parsl.dataflow.dflow INFO: Task 77 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898312/3b60610c-4806-4ea3-b6c0-4f2acb290708_calibrate_898312_2.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898312/3b60610c-4806-4ea3-b6c0-4f2acb290708_calibrate_898312_2.stderr parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898312/8c9728f4-fe0e-467c-9118-4c14df162c3d_characterizeImage_898312_2.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898312/8c9728f4-fe0e-467c-9118-4c14df162c3d_characterizeImage_898312_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 13 try 1 failed parsl.dataflow.dflow ERROR: Task 13 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry c72cc05cbe5d0f3f46a35f41069e2993 with result from task 13 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898377/769a2947-4530-42bc-9032-1647e8b4baa3_calibrate_898377_31.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898377/769a2947-4530-42bc-9032-1647e8b4baa3_calibrate_898377_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:47:17 proc_lsst.multi:146 [INFO] found job 17008924 in provider astro proc_lsst.multi INFO: found job 17008924 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 21 try 1 failed parsl.dataflow.dflow ERROR: Task 21 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 3c22e053a9a4ae24e4b55a94b408d45f with result from task 21 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898353/c49c449d-4f7a-4ba2-91fa-945b7c653983_calibrate_898353_31.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898353/c49c449d-4f7a-4ba2-91fa-945b7c653983_calibrate_898353_31.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 84 try 1 failed parsl.dataflow.dflow ERROR: Task 84 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 6c1c24bc55d30171015ad7bedde0f64d with result from task 84 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 85 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 85 try 0 failed parsl.dataflow.dflow INFO: Task 85 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898323/c08c80bc-0e27-4590-8628-fe4623972370_calibrate_898323_2.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898323/c08c80bc-0e27-4590-8628-fe4623972370_calibrate_898323_2.stderr parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898323/5133c2f4-b9fe-4bf7-9ff3-9a5d3df60280_characterizeImage_898323_2.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898323/5133c2f4-b9fe-4bf7-9ff3-9a5d3df60280_characterizeImage_898323_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:47:18 proc_lsst.multi:146 [INFO] found job 17008925 in provider astro proc_lsst.multi INFO: found job 17008925 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 1 try 1 failed parsl.dataflow.dflow ERROR: Task 1 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 66ab7b72d091b964779f664b0e32bb80 with result from task 1 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898344/5582ca13-75db-4a41-ab17-540aa0a69df1_calibrate_898344_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898344/5582ca13-75db-4a41-ab17-540aa0a69df1_calibrate_898344_31.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 7 try 1 failed parsl.dataflow.dflow ERROR: Task 7 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 7e4a79586e0a0d69281374a58e04e7d9 with result from task 7 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898342/dc2c0f58-ade0-4f10-9908-3da1526301af_calibrate_898342_31.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898342/dc2c0f58-ade0-4f10-9908-3da1526301af_calibrate_898342_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:47:20 proc_lsst.multi:146 [INFO] found job 17008926 in provider astro proc_lsst.multi INFO: found job 17008926 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 49 try 1 failed parsl.dataflow.dflow ERROR: Task 49 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 61298ebe490fd5b5bcb32fbd1ea462ad with result from task 49 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898327/f0e8c5c2-1e4e-4b94-97cf-0e192997e8f0_calibrate_898327_31.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898327/f0e8c5c2-1e4e-4b94-97cf-0e192997e8f0_calibrate_898327_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:47:21 proc_lsst.multi:146 [INFO] found job 17008927 in provider astro proc_lsst.multi INFO: found job 17008927 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 90 try 1 failed parsl.dataflow.dflow ERROR: Task 90 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 21c96c8f13668542a720b71501d6e6d6 with result from task 90 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 91 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 91 try 0 failed parsl.dataflow.dflow INFO: Task 91 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898321/0b7bcb6f-b5b4-4529-9009-6beafd7f9ba6_calibrate_898321_2.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898321/0b7bcb6f-b5b4-4529-9009-6beafd7f9ba6_calibrate_898321_2.stderr parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898321/b6b263c7-e416-493b-a529-87f8e2894e51_characterizeImage_898321_2.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898321/b6b263c7-e416-493b-a529-87f8e2894e51_characterizeImage_898321_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 26 try 1 failed parsl.dataflow.dflow ERROR: Task 26 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 404a922ad98363cbbcd02ddc8c89da0a with result from task 26 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898374/b2ab386a-7074-4980-a0ae-626e602344c7_calibrate_898374_31.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898374/b2ab386a-7074-4980-a0ae-626e602344c7_calibrate_898374_31.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 88 try 1 failed parsl.dataflow.dflow ERROR: Task 88 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 628def22a10cce152316023db3117880 with result from task 88 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 89 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 89 try 0 failed parsl.dataflow.dflow INFO: Task 89 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898302/69134ad6-0eda-472c-b22f-8a5e242ab065_calibrate_898302_2.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898302/69134ad6-0eda-472c-b22f-8a5e242ab065_calibrate_898302_2.stderr parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898302/0b03baeb-44ab-4f8f-b76b-8fe5966ddbd8_characterizeImage_898302_2.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898302/0b03baeb-44ab-4f8f-b76b-8fe5966ddbd8_characterizeImage_898302_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 5 try 1 failed parsl.dataflow.dflow ERROR: Task 5 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 7ab3a43ba4690db27a9d2acbf2b23156 with result from task 5 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898326/e8dc36e3-899c-4e77-95b7-12cea144d25c_calibrate_898326_31.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898326/e8dc36e3-899c-4e77-95b7-12cea144d25c_calibrate_898326_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:47:22 proc_lsst.multi:146 [INFO] found job 17008930 in provider astro proc_lsst.multi INFO: found job 17008930 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 92 try 1 failed parsl.dataflow.dflow ERROR: Task 92 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 261ae0ee6490810520ccbcac71c2312b with result from task 92 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 93 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 93 try 0 failed parsl.dataflow.dflow INFO: Task 93 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898297/f2c1d3a3-18bc-40d6-87d0-6ee4a233e5bf_calibrate_898297_2.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898297/f2c1d3a3-18bc-40d6-87d0-6ee4a233e5bf_calibrate_898297_2.stderr parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898297/669fdf04-b21b-4b07-b670-2caae0e92c0d_characterizeImage_898297_2.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898297/669fdf04-b21b-4b07-b670-2caae0e92c0d_characterizeImage_898297_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 94 try 1 failed parsl.dataflow.dflow ERROR: Task 94 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry f59722a9d505c1b91a9d5f565af0f069 with result from task 94 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 95 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 95 try 0 failed parsl.dataflow.dflow INFO: Task 95 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898332/ca763ea9-b2b7-4214-99a5-f7bc5eba0c66_calibrate_898332_2.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898332/ca763ea9-b2b7-4214-99a5-f7bc5eba0c66_calibrate_898332_2.stderr parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898332/b22b4ad9-5899-4a01-8fed-8564cd53ba37_characterizeImage_898332_2.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898332/b22b4ad9-5899-4a01-8fed-8564cd53ba37_characterizeImage_898332_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 22 try 1 failed parsl.dataflow.dflow ERROR: Task 22 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 483dcf17c6bb676a0286f433a5d209d2 with result from task 22 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898366/78353f3c-57e1-4237-8cc1-01f5ac43e0ee_calibrate_898366_31.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898366/78353f3c-57e1-4237-8cc1-01f5ac43e0ee_calibrate_898366_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:47:23 proc_lsst.multi:146 [INFO] found job 17008934 in provider astro proc_lsst.multi INFO: found job 17008934 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 43 try 1 failed parsl.dataflow.dflow ERROR: Task 43 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 68e95a36b0438dbfc00da1206a9ce4d5 with result from task 43 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898336/f5525908-304f-433e-9c72-7dddccdbfb5c_calibrate_898336_31.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898336/f5525908-304f-433e-9c72-7dddccdbfb5c_calibrate_898336_31.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 102 try 1 failed parsl.dataflow.dflow ERROR: Task 102 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 64e7fc366a0525f9369946feb5abbf09 with result from task 102 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 103 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 103 try 0 failed parsl.dataflow.dflow INFO: Task 103 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898365/8151fd08-a41b-48e9-8e0f-8232e1b1bd35_calibrate_898365_2.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898365/8151fd08-a41b-48e9-8e0f-8232e1b1bd35_calibrate_898365_2.stderr parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898365/041b194e-fef4-41c1-9660-e78ee710ec76_characterizeImage_898365_2.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898365/041b194e-fef4-41c1-9660-e78ee710ec76_characterizeImage_898365_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 100 try 1 failed parsl.dataflow.dflow ERROR: Task 100 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry fd409323414c1a1d0f6cd6241a0ac7a1 with result from task 100 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 101 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 101 try 0 failed parsl.dataflow.dflow INFO: Task 101 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898373/13c3008e-2329-4a51-a3b6-67dea4159eb3_calibrate_898373_2.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898373/13c3008e-2329-4a51-a3b6-67dea4159eb3_calibrate_898373_2.stderr parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898373/38ee9896-518b-48b2-a68f-de2b4b9536df_characterizeImage_898373_2.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898373/38ee9896-518b-48b2-a68f-de2b4b9536df_characterizeImage_898373_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 27 try 1 failed parsl.dataflow.dflow ERROR: Task 27 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 2154875a4ca8ccfdcf0b441f8ebe8127 with result from task 27 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898334/38bdbda3-4974-4bd8-b77c-ac6f04c45a29_calibrate_898334_31.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898334/38bdbda3-4974-4bd8-b77c-ac6f04c45a29_calibrate_898334_31.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 44 try 1 failed parsl.dataflow.dflow ERROR: Task 44 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 4069964c40da3c18bb04d283d4cd2a2e with result from task 44 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898324/f7963f80-1c9c-4bb7-97aa-466c1c22bdb9_calibrate_898324_31.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898324/f7963f80-1c9c-4bb7-97aa-466c1c22bdb9_calibrate_898324_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:47:25 proc_lsst.multi:146 [INFO] found job 17008943 in provider escience_rtx6k proc_lsst.multi INFO: found job 17008943 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008943,17008949,17008950,17008951,17008952' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 0 try 1 failed parsl.dataflow.dflow ERROR: Task 0 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 09066a0219e24fdbb84bb3def6e26179 with result from task 0 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898348/81898b87-3975-4957-9c7a-c96ad661712e_calibrate_898348_31.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898348/81898b87-3975-4957-9c7a-c96ad661712e_calibrate_898348_31.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 35 try 1 failed parsl.dataflow.dflow ERROR: Task 35 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry f7549c3b5ee5ec0cd39d9e522a784dd2 with result from task 35 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898307/a34da88c-ce76-49a3-9a04-93e927f2d5fa_calibrate_898307_31.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898307/a34da88c-ce76-49a3-9a04-93e927f2d5fa_calibrate_898307_31.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 82 try 1 failed parsl.dataflow.dflow ERROR: Task 82 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 654acdcd0a0eda691b5063d512516b42 with result from task 82 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 83 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 83 try 0 failed parsl.dataflow.dflow INFO: Task 83 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898358/c2c1ef65-6320-4c4d-b8b0-0879fff026b8_calibrate_898358_2.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898358/c2c1ef65-6320-4c4d-b8b0-0879fff026b8_calibrate_898358_2.stderr parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898358/e8ef6efb-7ecf-4b6d-a66f-16586850bca6_characterizeImage_898358_2.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898358/e8ef6efb-7ecf-4b6d-a66f-16586850bca6_characterizeImage_898358_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 96 try 1 failed parsl.dataflow.dflow ERROR: Task 96 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 4cd75d7ac7ccaa697b997bf5d33d8730 with result from task 96 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 97 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 97 try 0 failed parsl.dataflow.dflow INFO: Task 97 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898328/b59252af-393b-460c-8bf6-1c2af7f490f4_calibrate_898328_2.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898328/b59252af-393b-460c-8bf6-1c2af7f490f4_calibrate_898328_2.stderr parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898328/d604c1ba-966b-4f42-8457-f96d32e53430_characterizeImage_898328_2.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898328/d604c1ba-966b-4f42-8457-f96d32e53430_characterizeImage_898328_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 80 try 1 failed parsl.dataflow.dflow ERROR: Task 80 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 1493f5d7eb240fc798286fc52113f9e2 with result from task 80 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 81 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 81 try 0 failed parsl.dataflow.dflow INFO: Task 81 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898317/065aa81a-fcf4-46fc-9adf-99e62c3ad752_calibrate_898317_2.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898317/065aa81a-fcf4-46fc-9adf-99e62c3ad752_calibrate_898317_2.stderr parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898317/f42869ae-e0a4-4695-a94d-eae09f51d55c_characterizeImage_898317_2.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898317/f42869ae-e0a4-4695-a94d-eae09f51d55c_characterizeImage_898317_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 86 try 1 failed parsl.dataflow.dflow ERROR: Task 86 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 28c150fec6c19bf55aa1b4c8944de0f6 with result from task 86 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 87 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 87 try 0 failed parsl.dataflow.dflow INFO: Task 87 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898352/f3e1f4ab-90ba-4f85-9dfb-76cc84cb2d37_calibrate_898352_2.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898352/f3e1f4ab-90ba-4f85-9dfb-76cc84cb2d37_calibrate_898352_2.stderr parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898352/ea28b3c9-e970-4b1f-9393-bd5bf5dd9520_characterizeImage_898352_2.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898352/ea28b3c9-e970-4b1f-9393-bd5bf5dd9520_characterizeImage_898352_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008950 R 17008951 R 17008952 R 17008949 R 17008943 R parsl.providers.slurm.slurm DEBUG: Updating job 17008950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008943 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:47:26 proc_lsst.multi:146 [INFO] found job 17008949 in provider escience_rtx6k proc_lsst.multi INFO: found job 17008949 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008943,17008949,17008950,17008951,17008952' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 98 try 1 failed parsl.dataflow.dflow ERROR: Task 98 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 4e8b658cbaab8e531dc6f1e523d932e2 with result from task 98 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 99 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 99 try 0 failed parsl.dataflow.dflow INFO: Task 99 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898346/6d9560b5-ad26-4192-8c76-2b25b56ddbfb_calibrate_898346_2.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898346/6d9560b5-ad26-4192-8c76-2b25b56ddbfb_calibrate_898346_2.stderr parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898346/8b7896f6-d177-4d6d-977b-67c894156eee_characterizeImage_898346_2.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898346/8b7896f6-d177-4d6d-977b-67c894156eee_characterizeImage_898346_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 108 try 1 failed parsl.dataflow.dflow ERROR: Task 108 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 004a77f79aec5d3f5723722d59e73a05 with result from task 108 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 109 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 109 try 0 failed parsl.dataflow.dflow INFO: Task 109 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898330/6d5af2a1-6306-4122-b641-d972161973da_calibrate_898330_2.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898330/6d5af2a1-6306-4122-b641-d972161973da_calibrate_898330_2.stderr parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898330/60a380f1-e740-4766-baed-0adb4a25953e_characterizeImage_898330_2.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898330/60a380f1-e740-4766-baed-0adb4a25953e_characterizeImage_898330_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008950 R 17008951 R 17008952 R 17008949 R 17008943 R parsl.providers.slurm.slurm DEBUG: Updating job 17008950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008943 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:47:28 proc_lsst.multi:146 [INFO] found job 17008950 in provider escience_rtx6k proc_lsst.multi INFO: found job 17008950 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008943,17008949,17008950,17008951,17008952' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 28 try 1 failed parsl.dataflow.dflow ERROR: Task 28 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 9da04a8eed0e599dcf293179a3c308c4 with result from task 28 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898375/48301fc1-2890-4842-b21b-b4bda43648de_calibrate_898375_31.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898375/48301fc1-2890-4842-b21b-b4bda43648de_calibrate_898375_31.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 104 try 1 failed parsl.dataflow.dflow ERROR: Task 104 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry f00b1dd700441f6be9a0dd3f6d51e9a3 with result from task 104 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 105 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 105 try 0 failed parsl.dataflow.dflow INFO: Task 105 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898370/51a541a9-34e9-4fd2-8cd9-7474e94f8dd5_calibrate_898370_2.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898370/51a541a9-34e9-4fd2-8cd9-7474e94f8dd5_calibrate_898370_2.stderr parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898370/56990dd5-5747-4b9a-92a7-2fefb7fcd19b_characterizeImage_898370_2.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898370/56990dd5-5747-4b9a-92a7-2fefb7fcd19b_characterizeImage_898370_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 110 try 1 failed parsl.dataflow.dflow ERROR: Task 110 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry bc1139a2902a49b0759292637d4b2394 with result from task 110 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 111 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 111 try 0 failed parsl.dataflow.dflow INFO: Task 111 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898324/a7600b34-aadd-4f07-bb17-b4220f773446_calibrate_898324_2.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898324/a7600b34-aadd-4f07-bb17-b4220f773446_calibrate_898324_2.stderr parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898324/b42216a1-ed0d-4817-b690-a87e5f1b15a6_characterizeImage_898324_2.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898324/b42216a1-ed0d-4817-b690-a87e5f1b15a6_characterizeImage_898324_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 12 try 1 failed parsl.dataflow.dflow ERROR: Task 12 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry cad4018ac34f8c2db58928cbebfbe4e7 with result from task 12 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898349/353d01b1-13b9-47ef-8df8-b3c7f7d84717_calibrate_898349_31.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898349/353d01b1-13b9-47ef-8df8-b3c7f7d84717_calibrate_898349_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008950 R 17008951 R 17008952 R 17008949 R 17008943 R parsl.providers.slurm.slurm DEBUG: Updating job 17008950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008943 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:47:29 proc_lsst.multi:146 [INFO] found job 17008951 in provider escience_rtx6k proc_lsst.multi INFO: found job 17008951 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008943,17008949,17008950,17008951,17008952' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 15 try 1 failed parsl.dataflow.dflow ERROR: Task 15 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 4302c68bf760c49455c325a400d545a9 with result from task 15 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898341/b35cd10a-e0ae-4f4d-bded-3b07f4776216_calibrate_898341_31.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898341/b35cd10a-e0ae-4f4d-bded-3b07f4776216_calibrate_898341_31.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 114 try 1 failed parsl.dataflow.dflow ERROR: Task 114 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry d714e2248c9ad715ae07492a0747914d with result from task 114 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 115 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 115 try 0 failed parsl.dataflow.dflow INFO: Task 115 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898303/3edcfd71-b8f0-4fbd-a58c-59aac791d4f8_calibrate_898303_2.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898303/3edcfd71-b8f0-4fbd-a58c-59aac791d4f8_calibrate_898303_2.stderr parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898303/a533e21c-7e5e-45ca-aee7-0090ab76421b_characterizeImage_898303_2.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898303/a533e21c-7e5e-45ca-aee7-0090ab76421b_characterizeImage_898303_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 118 try 1 failed parsl.dataflow.dflow ERROR: Task 118 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 649fc220bc2bffba313897db8374eb97 with result from task 118 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 119 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 119 try 0 failed parsl.dataflow.dflow INFO: Task 119 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898345/880f4dd6-643a-4797-ad2b-2d4aafe7489b_calibrate_898345_2.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898345/880f4dd6-643a-4797-ad2b-2d4aafe7489b_calibrate_898345_2.stderr parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898345/cbf6c8ea-e38c-4321-85ca-0eac41820570_characterizeImage_898345_2.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898345/cbf6c8ea-e38c-4321-85ca-0eac41820570_characterizeImage_898345_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 36 try 0 failed parsl.dataflow.dflow INFO: Task 36 marked for retry parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898309/82e1efa9-7097-42b4-aede-e23e05ddbd81_calibrate_898309_61.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898309/82e1efa9-7097-42b4-aede-e23e05ddbd81_calibrate_898309_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 36 has memoization hash 61aee1dbd6fba070a7190edb6496165b parsl.dataflow.memoization INFO: Task 36 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b6a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 36 try 1 launched on executor multi with executor id 305 parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898309/82e1efa9-7097-42b4-aede-e23e05ddbd81_calibrate_898309_61.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898309/82e1efa9-7097-42b4-aede-e23e05ddbd81_calibrate_898309_61.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 34 try 1 failed parsl.dataflow.dflow ERROR: Task 34 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 5947a47f671ed559925e25b77189cefd with result from task 34 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898354/c244ec8e-369a-448b-a0ee-6e385648f4b1_calibrate_898354_31.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898354/c244ec8e-369a-448b-a0ee-6e385648f4b1_calibrate_898354_31.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 42 try 1 failed parsl.dataflow.dflow ERROR: Task 42 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry c4e1e65f38d0257accd555cc0a75ca7b with result from task 42 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898312/5cfe0bcb-2d52-4157-b2cd-bd5b702680b3_calibrate_898312_31.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898312/5cfe0bcb-2d52-4157-b2cd-bd5b702680b3_calibrate_898312_31.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 112 try 1 failed parsl.dataflow.dflow ERROR: Task 112 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 681c26047d26280295a0ebbee5ec6622 with result from task 112 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 113 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 113 try 0 failed parsl.dataflow.dflow INFO: Task 113 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898322/3a21a90f-1702-4ad0-8304-8b69c5f16e6f_calibrate_898322_2.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898322/3a21a90f-1702-4ad0-8304-8b69c5f16e6f_calibrate_898322_2.stderr parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898322/a639d512-32dc-4105-9ea8-df8022357a3b_characterizeImage_898322_2.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898322/a639d512-32dc-4105-9ea8-df8022357a3b_characterizeImage_898322_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 120 try 1 failed parsl.dataflow.dflow ERROR: Task 120 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 4f1959275519093a86082fc455f48e67 with result from task 120 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 121 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 121 try 0 failed parsl.dataflow.dflow INFO: Task 121 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898369/9243489d-f7ff-409f-97b5-24d85c27cbda_calibrate_898369_2.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898369/9243489d-f7ff-409f-97b5-24d85c27cbda_calibrate_898369_2.stderr parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898369/2fe593b4-90ae-4c32-b322-54407729f112_characterizeImage_898369_2.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898369/2fe593b4-90ae-4c32-b322-54407729f112_characterizeImage_898369_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008950 R 17008951 R 17008952 R 17008949 R 17008943 R parsl.providers.slurm.slurm DEBUG: Updating job 17008950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008943 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:47:30 proc_lsst.multi:146 [INFO] found job 17008952 in provider escience_rtx6k proc_lsst.multi INFO: found job 17008952 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008943,17008949,17008950,17008951,17008952' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 122 try 1 failed parsl.dataflow.dflow ERROR: Task 122 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry a5f8477f44414355d2876740a66b990a with result from task 122 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 123 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 123 try 0 failed parsl.dataflow.dflow INFO: Task 123 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898310/78b9f2df-4edc-47c0-8ec8-c34b7e2ce5b0_calibrate_898310_2.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898310/78b9f2df-4edc-47c0-8ec8-c34b7e2ce5b0_calibrate_898310_2.stderr parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898310/f0d0662d-41f6-4456-a487-ef4267af8fca_characterizeImage_898310_2.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898310/f0d0662d-41f6-4456-a487-ef4267af8fca_characterizeImage_898310_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 20 try 1 failed parsl.dataflow.dflow ERROR: Task 20 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 68bf16d6705930d3552ef85355e36440 with result from task 20 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898346/fa1671bf-7a1a-4e67-8fbe-bb109b12b349_calibrate_898346_31.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898346/fa1671bf-7a1a-4e67-8fbe-bb109b12b349_calibrate_898346_31.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 106 try 1 failed parsl.dataflow.dflow ERROR: Task 106 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 10e68e6a2238ca602f91fc1a6083824b with result from task 106 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 107 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 107 try 0 failed parsl.dataflow.dflow INFO: Task 107 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898356/7322897c-2f89-4664-8c9e-87a377db674b_calibrate_898356_2.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898356/7322897c-2f89-4664-8c9e-87a377db674b_calibrate_898356_2.stderr parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898356/1dde0e09-6994-4851-bffb-41645b1063f6_characterizeImage_898356_2.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898356/1dde0e09-6994-4851-bffb-41645b1063f6_characterizeImage_898356_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 124 try 1 failed parsl.dataflow.dflow ERROR: Task 124 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry d2ee61bdc2b93667ff095c09f615fac2 with result from task 124 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 125 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 125 try 0 failed parsl.dataflow.dflow INFO: Task 125 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898327/e2c37304-2cf1-4fcb-a63d-d4b01dc0fb03_calibrate_898327_2.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898327/e2c37304-2cf1-4fcb-a63d-d4b01dc0fb03_calibrate_898327_2.stderr parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898327/78d3fffa-c228-475d-941c-2f6d356c99b2_characterizeImage_898327_2.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898327/78d3fffa-c228-475d-941c-2f6d356c99b2_characterizeImage_898327_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 116 try 1 failed parsl.dataflow.dflow ERROR: Task 116 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 66d86101b452de8bf87175fc34ff50a4 with result from task 116 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 117 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 117 try 0 failed parsl.dataflow.dflow INFO: Task 117 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898371/8827504d-de61-478a-839a-aae7002e6139_calibrate_898371_2.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898371/8827504d-de61-478a-839a-aae7002e6139_calibrate_898371_2.stderr parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898371/6677da51-c7e3-4e99-a0d2-010a7e7f57b9_characterizeImage_898371_2.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898371/6677da51-c7e3-4e99-a0d2-010a7e7f57b9_characterizeImage_898371_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008950 R 17008951 R 17008952 R 17008949 R 17008943 R parsl.providers.slurm.slurm DEBUG: Updating job 17008950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008943 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:47:32 proc_lsst.multi:146 [INFO] found job 17008953 in provider escience_a40 proc_lsst.multi INFO: found job 17008953 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008953,17008954,17008961' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 130 try 1 failed parsl.dataflow.dflow ERROR: Task 130 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry e2d9b4af3199bf1a4af8b80b93803cbb with result from task 130 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 131 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 131 try 0 failed parsl.dataflow.dflow INFO: Task 131 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898342/5a2f533b-9fcf-40bb-92d6-23c895fe4dd5_calibrate_898342_2.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898342/5a2f533b-9fcf-40bb-92d6-23c895fe4dd5_calibrate_898342_2.stderr parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898342/e6cece1d-a943-4e40-ae51-8391141a889a_characterizeImage_898342_2.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898342/e6cece1d-a943-4e40-ae51-8391141a889a_characterizeImage_898342_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 33 try 0 failed parsl.dataflow.dflow INFO: Task 33 marked for retry parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898344/bf5bb1bb-5ab6-47f9-8e07-57808bd6e728_calibrate_898344_61.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898344/bf5bb1bb-5ab6-47f9-8e07-57808bd6e728_calibrate_898344_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 33 has memoization hash c88a06f6a83e0593b438fb1774dd2460 parsl.dataflow.memoization INFO: Task 33 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14b394b69d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17008961 R 17008954 R 17008953 R parsl.providers.slurm.slurm DEBUG: Updating job 17008961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008953 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:47:33 proc_lsst.multi:146 [INFO] found job 17008954 in provider escience_a40 proc_lsst.multi INFO: found job 17008954 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008953,17008954,17008961' parsl.dataflow.dflow INFO: Parsl task 33 try 1 launched on executor multi with executor id 306 parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898344/bf5bb1bb-5ab6-47f9-8e07-57808bd6e728_calibrate_898344_61.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898344/bf5bb1bb-5ab6-47f9-8e07-57808bd6e728_calibrate_898344_61.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 134 try 1 failed parsl.dataflow.dflow ERROR: Task 134 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry eef3adc3e008b9c140003faf6a90c2b7 with result from task 134 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 135 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 135 try 0 failed parsl.dataflow.dflow INFO: Task 135 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898293/fe8ae200-f644-44a1-9c6f-1fdafd6eea91_calibrate_898293_2.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898293/fe8ae200-f644-44a1-9c6f-1fdafd6eea91_calibrate_898293_2.stderr parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898293/3d3968e4-c012-4124-a509-f38e1e0d9fa8_characterizeImage_898293_2.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898293/3d3968e4-c012-4124-a509-f38e1e0d9fa8_characterizeImage_898293_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 136 try 1 failed parsl.dataflow.dflow ERROR: Task 136 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry a4b8335b5d19157dd964fbb1e42fd80a with result from task 136 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 137 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 137 try 0 failed parsl.dataflow.dflow INFO: Task 137 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898319/541682dd-1ec4-48b5-b1b4-a8736ec2c071_calibrate_898319_2.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898319/541682dd-1ec4-48b5-b1b4-a8736ec2c071_calibrate_898319_2.stderr parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898319/97c032f7-fed0-48f5-98c5-8ef97841e90c_characterizeImage_898319_2.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898319/97c032f7-fed0-48f5-98c5-8ef97841e90c_characterizeImage_898319_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 140 try 1 failed parsl.dataflow.dflow ERROR: Task 140 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry a537e402d903b036ed90a11e2e1b344e with result from task 140 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 141 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 141 try 0 failed parsl.dataflow.dflow INFO: Task 141 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898318/0ce698e8-7580-487d-b3b1-e22987df2975_calibrate_898318_2.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898318/0ce698e8-7580-487d-b3b1-e22987df2975_calibrate_898318_2.stderr parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898318/fc5dc045-bf0e-43d6-95e3-1b7b0a0635f7_characterizeImage_898318_2.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898318/fc5dc045-bf0e-43d6-95e3-1b7b0a0635f7_characterizeImage_898318_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 138 try 1 failed parsl.dataflow.dflow ERROR: Task 138 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry ff5b909ecfc00a8ede54916c785b683f with result from task 138 parsl.providers.slurm.slurm DEBUG: squeue returned 17008961 R 17008954 R 17008953 R parsl.providers.slurm.slurm DEBUG: Updating job 17008961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008953 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:47:35 proc_lsst.multi:146 [INFO] found job 17008961 in provider escience_a40 proc_lsst.multi INFO: found job 17008961 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008953,17008954,17008961' parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 139 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 139 try 0 failed parsl.dataflow.dflow INFO: Task 139 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898339/af44c281-abdb-408a-a3ad-23277055c94e_calibrate_898339_2.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898339/af44c281-abdb-408a-a3ad-23277055c94e_calibrate_898339_2.stderr parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898339/91647e66-469e-416e-a268-f8eed54c914e_characterizeImage_898339_2.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898339/91647e66-469e-416e-a268-f8eed54c914e_characterizeImage_898339_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 142 try 1 failed parsl.dataflow.dflow ERROR: Task 142 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry c3c2e67415f37b774d20ed2a974e664b with result from task 142 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 143 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 143 try 0 failed parsl.dataflow.dflow INFO: Task 143 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898287/20427908-1b2a-4cec-be66-b36fd5f3d033_calibrate_898287_2.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898287/20427908-1b2a-4cec-be66-b36fd5f3d033_calibrate_898287_2.stderr parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898287/3fded289-8eed-4ce8-b353-f1e8205743a1_characterizeImage_898287_2.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898287/3fded289-8eed-4ce8-b353-f1e8205743a1_characterizeImage_898287_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 152 try 1 failed parsl.dataflow.dflow ERROR: Task 152 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 407875da43491a938cd49d59b18c6273 with result from task 152 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 153 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 153 try 0 failed parsl.dataflow.dflow INFO: Task 153 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898354/3023ab95-b9e1-4d40-9907-172add89c5f3_calibrate_898354_2.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898354/3023ab95-b9e1-4d40-9907-172add89c5f3_calibrate_898354_2.stderr parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898354/0704352b-dd37-4eb0-b3ca-3928ab62298f_characterizeImage_898354_2.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898354/0704352b-dd37-4eb0-b3ca-3928ab62298f_characterizeImage_898354_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 150 try 1 failed parsl.dataflow.dflow ERROR: Task 150 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry e0f7a4f2a8ae6516d8b754d10ea1c74d with result from task 150 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 151 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 151 try 0 failed parsl.dataflow.dflow INFO: Task 151 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898344/46b9d134-e486-4289-8d7e-7abeab194e27_calibrate_898344_2.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898344/46b9d134-e486-4289-8d7e-7abeab194e27_calibrate_898344_2.stderr parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898344/ad25b03a-d309-4099-a4c8-f0db38b05219_characterizeImage_898344_2.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898344/ad25b03a-d309-4099-a4c8-f0db38b05219_characterizeImage_898344_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 144 try 1 failed parsl.dataflow.dflow ERROR: Task 144 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 9c97e3fa9e28b23d513b63c136164642 with result from task 144 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 145 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 145 try 0 failed parsl.dataflow.dflow INFO: Task 145 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898337/585832a8-803c-41c3-b53c-18f20240843e_calibrate_898337_2.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898337/585832a8-803c-41c3-b53c-18f20240843e_calibrate_898337_2.stderr parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898337/3bedf328-bece-48f2-a146-6fdd66b27359_characterizeImage_898337_2.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898337/3bedf328-bece-48f2-a146-6fdd66b27359_characterizeImage_898337_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 154 try 1 failed parsl.dataflow.dflow ERROR: Task 154 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry eb57123a91a4dde1df4fd39a8b6f90d8 with result from task 154 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 155 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 155 try 0 failed parsl.dataflow.dflow INFO: Task 155 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898294/d0568f78-e266-4ce6-8c61-5ac44f7a566f_calibrate_898294_2.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898294/d0568f78-e266-4ce6-8c61-5ac44f7a566f_calibrate_898294_2.stderr parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898294/f5b9f324-1334-4e2e-a191-f3b12aaa5965_characterizeImage_898294_2.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898294/f5b9f324-1334-4e2e-a191-f3b12aaa5965_characterizeImage_898294_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 148 try 1 failed parsl.dataflow.dflow ERROR: Task 148 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 76e251355064399dc11343e930b21e23 with result from task 148 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 149 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 149 try 0 failed parsl.dataflow.dflow INFO: Task 149 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898360/b7f4cc72-1982-456f-9411-6004cd3e1ffb_calibrate_898360_2.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898360/b7f4cc72-1982-456f-9411-6004cd3e1ffb_calibrate_898360_2.stderr parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898360/6f8c9e81-9a6b-4554-8d15-ce766f7c0e51_characterizeImage_898360_2.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898360/6f8c9e81-9a6b-4554-8d15-ce766f7c0e51_characterizeImage_898360_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 156 try 1 failed parsl.dataflow.dflow ERROR: Task 156 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry b67bc97637cec3612a5869fcc9cf54f0 with result from task 156 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 157 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 157 try 0 failed parsl.dataflow.dflow INFO: Task 157 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898286/46fafd71-0ac7-443b-8b98-c25ee380cb9f_calibrate_898286_2.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898286/46fafd71-0ac7-443b-8b98-c25ee380cb9f_calibrate_898286_2.stderr parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898286/298e4d3d-7adc-4228-b469-702d4902d5d1_characterizeImage_898286_2.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898286/298e4d3d-7adc-4228-b469-702d4902d5d1_characterizeImage_898286_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 162 try 1 failed parsl.dataflow.dflow ERROR: Task 162 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry ea6a56b7ace35c002723f8d6242d2385 with result from task 162 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 163 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 163 try 0 failed parsl.dataflow.dflow INFO: Task 163 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898315/79f6c4ec-3fd1-41a8-a3dd-ec16746b240a_calibrate_898315_2.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898315/79f6c4ec-3fd1-41a8-a3dd-ec16746b240a_calibrate_898315_2.stderr parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898315/e2b2c7ea-f52d-42e1-a0fd-3e90ac60d7e2_characterizeImage_898315_2.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898315/e2b2c7ea-f52d-42e1-a0fd-3e90ac60d7e2_characterizeImage_898315_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 164 try 1 failed parsl.dataflow.dflow ERROR: Task 164 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 308fe0cd5112c0272d33c1f37c2dfe89 with result from task 164 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 165 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 165 try 0 failed parsl.dataflow.dflow INFO: Task 165 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898350/d1a16a18-a2e2-41a2-a0a6-79bac5545f39_calibrate_898350_2.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898350/d1a16a18-a2e2-41a2-a0a6-79bac5545f39_calibrate_898350_2.stderr parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898350/6e8de21e-cc46-48e9-babd-fad71dbccbc4_characterizeImage_898350_2.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898350/6e8de21e-cc46-48e9-babd-fad71dbccbc4_characterizeImage_898350_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 126 try 1 failed parsl.dataflow.dflow ERROR: Task 126 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry b38f6a39639aa9b0ee832c766689c8fa with result from task 126 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 127 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 127 try 0 failed parsl.dataflow.dflow INFO: Task 127 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898295/c30af083-3f02-4a70-aa37-d05856ce29c7_calibrate_898295_2.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898295/c30af083-3f02-4a70-aa37-d05856ce29c7_calibrate_898295_2.stderr parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898295/dfde6b77-0194-4de1-b802-86a134975378_characterizeImage_898295_2.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898295/dfde6b77-0194-4de1-b802-86a134975378_characterizeImage_898295_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 128 try 1 failed parsl.dataflow.dflow ERROR: Task 128 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry e714a8c3f307643a6b4d1d75d6a45e73 with result from task 128 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 129 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 129 try 0 failed parsl.dataflow.dflow INFO: Task 129 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898376/40cee5de-cb9a-4ea2-a9c7-8752ee656ff5_calibrate_898376_2.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898376/40cee5de-cb9a-4ea2-a9c7-8752ee656ff5_calibrate_898376_2.stderr parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898376/b14bc756-1590-4a0e-b075-74eecaae062f_characterizeImage_898376_2.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898376/b14bc756-1590-4a0e-b075-74eecaae062f_characterizeImage_898376_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 160 try 1 failed parsl.dataflow.dflow ERROR: Task 160 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 45bc3cff8c8d52a100a544bcbdac6f1e with result from task 160 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 161 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 161 try 0 failed parsl.dataflow.dflow INFO: Task 161 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898307/23fe19f8-99e2-4eea-aa3b-b46ae1274a83_calibrate_898307_2.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898307/23fe19f8-99e2-4eea-aa3b-b46ae1274a83_calibrate_898307_2.stderr parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898307/600d84f7-6e60-4406-8464-e4a9809afada_characterizeImage_898307_2.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898307/600d84f7-6e60-4406-8464-e4a9809afada_characterizeImage_898307_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 132 try 1 failed parsl.dataflow.dflow ERROR: Task 132 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 46027d2f5a228a5f097bec7cb9256df5 with result from task 132 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 133 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 133 try 0 failed parsl.dataflow.dflow INFO: Task 133 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898333/77311396-8858-4911-bbfd-b043314449ba_calibrate_898333_2.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898333/77311396-8858-4911-bbfd-b043314449ba_calibrate_898333_2.stderr parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898333/14edeee9-1368-4638-9bd8-cad4be9f250f_characterizeImage_898333_2.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898333/14edeee9-1368-4638-9bd8-cad4be9f250f_characterizeImage_898333_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 146 try 1 failed parsl.dataflow.dflow ERROR: Task 146 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 2b0511c6c1b4436dd88b008f34f8d4a2 with result from task 146 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 147 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 147 try 0 failed parsl.dataflow.dflow INFO: Task 147 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898362/f78ad475-9d82-4df6-9931-1628afe399fe_calibrate_898362_2.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898362/f78ad475-9d82-4df6-9931-1628afe399fe_calibrate_898362_2.stderr parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898362/273324db-4d70-46d8-a1a7-f4b198e1783f_characterizeImage_898362_2.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898362/273324db-4d70-46d8-a1a7-f4b198e1783f_characterizeImage_898362_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 166 try 1 failed parsl.dataflow.dflow ERROR: Task 166 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 9e37b0df012723d5c38d6fd39629e374 with result from task 166 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 167 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 167 try 0 failed parsl.dataflow.dflow INFO: Task 167 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898300/c469db81-7eb2-4665-89b4-093c259eda5b_calibrate_898300_2.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898300/c469db81-7eb2-4665-89b4-093c259eda5b_calibrate_898300_2.stderr parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898300/b94fef20-f629-47af-93e6-c9f2a3a9d58e_characterizeImage_898300_2.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898300/b94fef20-f629-47af-93e6-c9f2a3a9d58e_characterizeImage_898300_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 9 try 1 failed parsl.dataflow.dflow ERROR: Task 9 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 6bdcb389812a683b49c783b6c448d66b with result from task 9 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898382/4ab7025a-27b7-4b84-91f5-ad14420c245e_calibrate_898382_7.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898382/4ab7025a-27b7-4b84-91f5-ad14420c245e_calibrate_898382_7.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 158 try 1 failed parsl.dataflow.dflow ERROR: Task 158 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry a24c7ea12e2c2d17472c2e285c904e3c with result from task 158 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 159 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 159 try 0 failed parsl.dataflow.dflow INFO: Task 159 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898368/dbb678c4-44a8-4a91-b2d3-368a395b70e5_calibrate_898368_2.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898368/dbb678c4-44a8-4a91-b2d3-368a395b70e5_calibrate_898368_2.stderr parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898368/402adb10-54c3-4a78-9606-4fcca2726950_characterizeImage_898368_2.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898368/402adb10-54c3-4a78-9606-4fcca2726950_characterizeImage_898368_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 174 try 1 failed parsl.dataflow.dflow ERROR: Task 174 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 53d95f458579079a7083b5d9467898cf with result from task 174 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 175 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 175 try 0 failed parsl.dataflow.dflow INFO: Task 175 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898325/d4fdba22-e503-4f40-819c-c0b63fca3883_calibrate_898325_2.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898325/d4fdba22-e503-4f40-819c-c0b63fca3883_calibrate_898325_2.stderr parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898325/a1d83ff3-a248-4b6d-a09b-46c0ec2b1db9_characterizeImage_898325_2.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898325/a1d83ff3-a248-4b6d-a09b-46c0ec2b1db9_characterizeImage_898325_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 178 try 1 failed parsl.dataflow.dflow ERROR: Task 178 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 5e6ed418d182a60ebe3b6707af1f3fb7 with result from task 178 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 179 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 179 try 0 failed parsl.dataflow.dflow INFO: Task 179 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898316/86cb7c21-ff11-43ef-b6c3-a3394f36c2f8_calibrate_898316_2.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898316/86cb7c21-ff11-43ef-b6c3-a3394f36c2f8_calibrate_898316_2.stderr parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898316/859c2f84-b5d9-4766-8e01-03ce74cd180a_characterizeImage_898316_2.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898316/859c2f84-b5d9-4766-8e01-03ce74cd180a_characterizeImage_898316_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 180 try 1 failed parsl.dataflow.dflow ERROR: Task 180 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 83183bddadbe819dd868ad0180af8efb with result from task 180 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 181 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 181 try 0 failed parsl.dataflow.dflow INFO: Task 181 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898340/6394d360-7b43-44b4-85bf-e8c5239a94dc_calibrate_898340_2.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898340/6394d360-7b43-44b4-85bf-e8c5239a94dc_calibrate_898340_2.stderr parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898340/d4cf30dc-0e16-4a99-b41a-89a591835063_characterizeImage_898340_2.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898340/d4cf30dc-0e16-4a99-b41a-89a591835063_characterizeImage_898340_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 176 try 1 failed parsl.dataflow.dflow ERROR: Task 176 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 789a3cf4fbdb6926020ca79be33c4a3b with result from task 176 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 177 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 177 try 0 failed parsl.dataflow.dflow INFO: Task 177 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898289/3c54a6c2-9c87-4992-a025-d4efc2374225_calibrate_898289_2.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898289/3c54a6c2-9c87-4992-a025-d4efc2374225_calibrate_898289_2.stderr parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898289/826cbbf6-3f98-46d6-8550-55ac105a4087_characterizeImage_898289_2.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898289/826cbbf6-3f98-46d6-8550-55ac105a4087_characterizeImage_898289_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 172 try 1 failed parsl.dataflow.dflow ERROR: Task 172 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 82a7632dea989fb59b9debf483e8f257 with result from task 172 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 173 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 173 try 0 failed parsl.dataflow.dflow INFO: Task 173 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898291/752be6e9-c81b-4ce1-9c4b-a31daf8224c0_calibrate_898291_2.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898291/752be6e9-c81b-4ce1-9c4b-a31daf8224c0_calibrate_898291_2.stderr parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898291/3f08e496-3c1c-4e7b-81b8-e00bfd49915c_characterizeImage_898291_2.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898291/3f08e496-3c1c-4e7b-81b8-e00bfd49915c_characterizeImage_898291_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 188 try 1 failed parsl.dataflow.dflow ERROR: Task 188 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 3732fc702b75d329c601489899e3f640 with result from task 188 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 189 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 189 try 0 failed parsl.dataflow.dflow INFO: Task 189 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898304/bdf5afb0-c125-4106-b3bf-f97ba5ba2c26_calibrate_898304_2.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898304/bdf5afb0-c125-4106-b3bf-f97ba5ba2c26_calibrate_898304_2.stderr parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898304/f4fe1812-be36-41cd-8197-d661a0fcd8c9_characterizeImage_898304_2.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898304/f4fe1812-be36-41cd-8197-d661a0fcd8c9_characterizeImage_898304_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 168 try 1 failed parsl.dataflow.dflow ERROR: Task 168 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry b04f9d26fa7a7033635a9284cede03de with result from task 168 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 169 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 169 try 0 failed parsl.dataflow.dflow INFO: Task 169 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898353/4bd004ca-515e-4468-903a-06d009e0c8e5_calibrate_898353_2.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898353/4bd004ca-515e-4468-903a-06d009e0c8e5_calibrate_898353_2.stderr parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898353/8b3003ab-d982-40fb-9a29-d42d3753c93a_characterizeImage_898353_2.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898353/8b3003ab-d982-40fb-9a29-d42d3753c93a_characterizeImage_898353_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 194 try 1 failed parsl.dataflow.dflow ERROR: Task 194 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry c862e51217a6299650d85a3e39d0d8d7 with result from task 194 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 195 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 195 try 0 failed parsl.dataflow.dflow INFO: Task 195 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898366/960b49f1-c2a2-46c4-9d86-fb668353d85f_calibrate_898366_2.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898366/960b49f1-c2a2-46c4-9d86-fb668353d85f_calibrate_898366_2.stderr parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898366/883e1fdb-0061-4803-a77b-f6a737d0eb4d_characterizeImage_898366_2.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898366/883e1fdb-0061-4803-a77b-f6a737d0eb4d_characterizeImage_898366_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 170 try 1 failed parsl.dataflow.dflow ERROR: Task 170 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry bb3d28e661ccb7a9e2497ea69d198be1 with result from task 170 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 171 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 171 try 0 failed parsl.dataflow.dflow INFO: Task 171 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898381/59741d8d-13ce-4cc0-9eca-0b61914bff32_calibrate_898381_2.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898381/59741d8d-13ce-4cc0-9eca-0b61914bff32_calibrate_898381_2.stderr parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898381/def23eee-f468-417e-a13d-2c3590830f1a_characterizeImage_898381_2.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898381/def23eee-f468-417e-a13d-2c3590830f1a_characterizeImage_898381_2.stderr parsl.channels.local.local WARNING: Execution of command 'squeue --noheader --format='%i %t' --job '17008953,17008954,17008961'' failed due to Command 'squeue --noheader --format='%i %t' --job '17008953,17008954,17008961'' timed out after 10 seconds parsl.utils ERROR: Callback threw an exception - logging and proceeding anyway Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/utils.py", line 341, in make_callback self.callback(*self.cb_args) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/job_status_poller.py", line 114, in poll self._update_state() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/job_status_poller.py", line 121, in _update_state item.poll(now) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/job_status_poller.py", line 47, in poll self._status = self._executor.status() ^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/executors/status_handling.py", line 109, in status status = self._make_status_dict(block_ids, self._provider.status(job_ids)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/gscratch/dirac/shared/opt/proc_lsst/python/proc_lsst/multi.py", line 190, in status statuses += self.providers[provider].status([job_id]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/providers/cluster_provider.py", line 142, in status self._status() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/providers/slurm/slurm.py", line 157, in _status retcode, stdout, stderr = self.execute_wait(cmd) ^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/providers/cluster_provider.py", line 83, in execute_wait return self.channel.execute_wait(cmd, t) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/channels/local/local.py", line 71, in execute_wait (stdout, stderr) = proc.communicate(timeout=walltime) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/subprocess.py", line 1209, in communicate stdout, stderr = self._communicate(input, endtime, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/subprocess.py", line 2114, in _communicate self._check_timeout(endtime, orig_timeout, stdout, stderr) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/subprocess.py", line 1253, in _check_timeout raise TimeoutExpired( subprocess.TimeoutExpired: Command 'squeue --noheader --format='%i %t' --job '17008953,17008954,17008961'' timed out after 10 seconds 2024-03-13 08:47:45 proc_lsst.multi:146 [INFO] found job 34092 in provider local proc_lsst.multi INFO: found job 34092 in provider local 2024-03-13 08:47:45 proc_lsst.multi:146 [INFO] found job 17008896 in provider astro proc_lsst.multi INFO: found job 17008896 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 184 try 1 failed parsl.dataflow.dflow ERROR: Task 184 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry c3b73f90ebf119d57db71ad1aae6ce4e with result from task 184 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 185 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 185 try 0 failed parsl.dataflow.dflow INFO: Task 185 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898374/7a905829-b5b7-427e-b3ff-a0a69d08538e_calibrate_898374_2.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898374/7a905829-b5b7-427e-b3ff-a0a69d08538e_calibrate_898374_2.stderr parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898374/7d868f31-aea1-4df3-bca4-b4f657a601a8_characterizeImage_898374_2.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898374/7d868f31-aea1-4df3-bca4-b4f657a601a8_characterizeImage_898374_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 54 try 1 failed parsl.dataflow.dflow ERROR: Task 54 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 71a530e2f28b0764e94bcbd4c6ecbd86 with result from task 54 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898318/b13456ab-f9fa-4fb6-81b1-66a42b8b3c13_calibrate_898318_31.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898318/b13456ab-f9fa-4fb6-81b1-66a42b8b3c13_calibrate_898318_31.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 196 try 1 failed parsl.dataflow.dflow ERROR: Task 196 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 7c164b614258f56e5d1018602d2c6b22 with result from task 196 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 197 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 197 try 0 failed parsl.dataflow.dflow INFO: Task 197 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898313/4fb77d6b-526f-405e-8641-d0baab9bf56f_calibrate_898313_2.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898313/4fb77d6b-526f-405e-8641-d0baab9bf56f_calibrate_898313_2.stderr parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898313/8efe628c-fbcb-4620-b8a1-da84903ac46a_characterizeImage_898313_2.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898313/8efe628c-fbcb-4620-b8a1-da84903ac46a_characterizeImage_898313_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 182 try 1 failed parsl.dataflow.dflow ERROR: Task 182 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 187009d2c954513b50496652950b3e61 with result from task 182 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 183 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 183 try 0 failed parsl.dataflow.dflow INFO: Task 183 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898320/2383b152-be14-48aa-89f6-ba07db94bb53_calibrate_898320_2.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898320/2383b152-be14-48aa-89f6-ba07db94bb53_calibrate_898320_2.stderr parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898320/0d271a51-6393-4e3c-8de9-ccd83369b17f_characterizeImage_898320_2.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898320/0d271a51-6393-4e3c-8de9-ccd83369b17f_characterizeImage_898320_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 52 try 1 failed parsl.dataflow.dflow ERROR: Task 52 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 546bcd619bfed86dbc58546373cc23c9 with result from task 52 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898352/3a68dde5-3364-44e6-9bda-ed56a9f9eaac_calibrate_898352_31.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898352/3a68dde5-3364-44e6-9bda-ed56a9f9eaac_calibrate_898352_31.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 198 try 1 failed parsl.dataflow.dflow ERROR: Task 198 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 9860e33d4639157e87123b78bcea19b9 with result from task 198 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 199 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 199 try 0 failed parsl.dataflow.dflow INFO: Task 199 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898382/77fce85b-1f95-4b08-8961-bcee05dca6b1_calibrate_898382_2.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898382/77fce85b-1f95-4b08-8961-bcee05dca6b1_calibrate_898382_2.stderr parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898382/fdd9caee-a9d8-47b5-8d09-8b6e064291a9_characterizeImage_898382_2.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898382/fdd9caee-a9d8-47b5-8d09-8b6e064291a9_characterizeImage_898382_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:47:46 proc_lsst.multi:146 [INFO] found job 17008898 in provider astro proc_lsst.multi INFO: found job 17008898 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 208 try 1 failed parsl.dataflow.dflow ERROR: Task 208 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 87aac7584d5218dd9b19f278ccc6e0d5 with result from task 208 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 209 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 209 try 0 failed parsl.dataflow.dflow INFO: Task 209 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898378/6c0edc2c-0ecc-4ee6-83c3-a83759a5edaa_calibrate_898378_2.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898378/6c0edc2c-0ecc-4ee6-83c3-a83759a5edaa_calibrate_898378_2.stderr parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898378/0995fe10-ea29-4d67-a721-fb8b6cd22e66_characterizeImage_898378_2.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898378/0995fe10-ea29-4d67-a721-fb8b6cd22e66_characterizeImage_898378_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 206 try 1 failed parsl.dataflow.dflow ERROR: Task 206 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 47cbca9dbf88bdcdc02765303207aee9 with result from task 206 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 207 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 207 try 0 failed parsl.dataflow.dflow INFO: Task 207 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898292/d71cffd0-7e5f-428f-80ca-5be53d33db2f_calibrate_898292_2.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898292/d71cffd0-7e5f-428f-80ca-5be53d33db2f_calibrate_898292_2.stderr parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898292/3ca39bab-f6c2-4bd5-9b58-06747c252ce8_characterizeImage_898292_2.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898292/3ca39bab-f6c2-4bd5-9b58-06747c252ce8_characterizeImage_898292_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 210 try 1 failed parsl.dataflow.dflow ERROR: Task 210 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 37c00a91549aa1477fe31946f97a7c5f with result from task 210 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 211 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 211 try 0 failed parsl.dataflow.dflow INFO: Task 211 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898379/417bbbfd-fead-4000-bed5-7c54df8c1e92_calibrate_898379_2.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898379/417bbbfd-fead-4000-bed5-7c54df8c1e92_calibrate_898379_2.stderr parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898379/64202fb7-6336-4e9f-8200-02d1651dce8f_characterizeImage_898379_2.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898379/64202fb7-6336-4e9f-8200-02d1651dce8f_characterizeImage_898379_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:47:47 proc_lsst.multi:146 [INFO] found job 17008899 in provider astro proc_lsst.multi INFO: found job 17008899 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 202 try 1 failed parsl.dataflow.dflow ERROR: Task 202 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 5856865b095f60519def504b9848d2e8 with result from task 202 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 203 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 203 try 0 failed parsl.dataflow.dflow INFO: Task 203 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898305/aed95052-e101-48c8-a7d7-4282d04618e9_calibrate_898305_2.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898305/aed95052-e101-48c8-a7d7-4282d04618e9_calibrate_898305_2.stderr parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898305/6eb2b7de-5132-4dab-852e-cf7330151622_characterizeImage_898305_2.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898305/6eb2b7de-5132-4dab-852e-cf7330151622_characterizeImage_898305_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:47:49 proc_lsst.multi:146 [INFO] found job 17008900 in provider astro proc_lsst.multi INFO: found job 17008900 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 190 try 1 failed parsl.dataflow.dflow ERROR: Task 190 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 3492ba1a18c1958aa8dd61434111c302 with result from task 190 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 191 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 191 try 0 failed parsl.dataflow.dflow INFO: Task 191 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898341/c7bde3bd-5d4d-4975-9d08-70c031087c06_calibrate_898341_2.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898341/c7bde3bd-5d4d-4975-9d08-70c031087c06_calibrate_898341_2.stderr parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898341/1ff0f251-6bb7-4969-aad8-00b357877c1c_characterizeImage_898341_2.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898341/1ff0f251-6bb7-4969-aad8-00b357877c1c_characterizeImage_898341_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 204 try 1 failed parsl.dataflow.dflow ERROR: Task 204 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 24d2c9716569392e6d623f504981bd3b with result from task 204 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 205 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 205 try 0 failed parsl.dataflow.dflow INFO: Task 205 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898308/faacf823-5027-46e9-8c73-5db6b513cd7a_calibrate_898308_2.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898308/faacf823-5027-46e9-8c73-5db6b513cd7a_calibrate_898308_2.stderr parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898308/243ba183-700c-4b37-9a39-4b8717e68bde_characterizeImage_898308_2.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898308/243ba183-700c-4b37-9a39-4b8717e68bde_characterizeImage_898308_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 218 try 1 failed parsl.dataflow.dflow ERROR: Task 218 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 7cf33828fc86323e1122578488096e86 with result from task 218 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 219 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 219 try 0 failed parsl.dataflow.dflow INFO: Task 219 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898372/56f95031-4899-4e4c-be73-2e46bf70e57a_calibrate_898372_2.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898372/56f95031-4899-4e4c-be73-2e46bf70e57a_calibrate_898372_2.stderr parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898372/92791b56-4fa4-4dbb-928e-3fbe354d21b6_characterizeImage_898372_2.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898372/92791b56-4fa4-4dbb-928e-3fbe354d21b6_characterizeImage_898372_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 212 try 1 failed parsl.dataflow.dflow ERROR: Task 212 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 5dcdaaa9962234e2d4da66b378702105 with result from task 212 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 213 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 213 try 0 failed parsl.dataflow.dflow INFO: Task 213 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898336/bff16e6c-1219-4a02-a114-88e8a84df30f_calibrate_898336_2.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898336/bff16e6c-1219-4a02-a114-88e8a84df30f_calibrate_898336_2.stderr parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898336/59045ea5-e278-4d7d-a033-56463a2f5fac_characterizeImage_898336_2.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898336/59045ea5-e278-4d7d-a033-56463a2f5fac_characterizeImage_898336_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:47:50 proc_lsst.multi:146 [INFO] found job 17008901 in provider astro proc_lsst.multi INFO: found job 17008901 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 222 try 1 failed parsl.dataflow.dflow ERROR: Task 222 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry b6a5488867a8c941e81fba6ca799cbd6 with result from task 222 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 223 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 223 try 0 failed parsl.dataflow.dflow INFO: Task 223 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898299/30a5a385-6fbb-43aa-aa44-08ab00758669_calibrate_898299_2.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898299/30a5a385-6fbb-43aa-aa44-08ab00758669_calibrate_898299_2.stderr parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898299/a64d3126-e727-422d-a274-393d2146b227_characterizeImage_898299_2.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898299/a64d3126-e727-422d-a274-393d2146b227_characterizeImage_898299_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:47:51 proc_lsst.multi:146 [INFO] found job 17008902 in provider astro proc_lsst.multi INFO: found job 17008902 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 230 try 1 failed parsl.dataflow.dflow ERROR: Task 230 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry c7b4b8af22ecad1d67932da5957e2f6c with result from task 230 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 231 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 231 try 0 failed parsl.dataflow.dflow INFO: Task 231 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898367/d3ce4be8-7307-4fa7-813c-5356a18f1336_calibrate_898367_2.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898367/d3ce4be8-7307-4fa7-813c-5356a18f1336_calibrate_898367_2.stderr parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898367/1d9ca05f-cb75-4481-9d04-7f4a93801fb7_characterizeImage_898367_2.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898367/1d9ca05f-cb75-4481-9d04-7f4a93801fb7_characterizeImage_898367_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 224 try 1 failed parsl.dataflow.dflow ERROR: Task 224 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 18f9bb934b3848c102c6446fb94941ad with result from task 224 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 225 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 225 try 0 failed parsl.dataflow.dflow INFO: Task 225 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898359/d4328b52-0830-4cd6-902e-028442c7ad7a_calibrate_898359_2.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898359/d4328b52-0830-4cd6-902e-028442c7ad7a_calibrate_898359_2.stderr parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898359/65636a59-80fb-47c1-8f98-43c4f999b198_characterizeImage_898359_2.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898359/65636a59-80fb-47c1-8f98-43c4f999b198_characterizeImage_898359_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 214 try 1 failed parsl.dataflow.dflow ERROR: Task 214 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 9e891114e331dc0777c030114d61ec77 with result from task 214 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 215 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 215 try 0 failed parsl.dataflow.dflow INFO: Task 215 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898351/96845562-6a2c-4231-a3b1-19d2abe37581_calibrate_898351_2.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898351/96845562-6a2c-4231-a3b1-19d2abe37581_calibrate_898351_2.stderr parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898351/8862f32c-0af5-4ab1-84b5-911135b20c05_characterizeImage_898351_2.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898351/8862f32c-0af5-4ab1-84b5-911135b20c05_characterizeImage_898351_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 226 try 1 failed parsl.dataflow.dflow ERROR: Task 226 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 70e29d359f3a79d340340ba3825a7d7c with result from task 226 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 227 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 227 try 0 failed parsl.dataflow.dflow INFO: Task 227 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898329/b6a5ce21-1aa7-4b3b-ac92-7fc1f940e938_calibrate_898329_2.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898329/b6a5ce21-1aa7-4b3b-ac92-7fc1f940e938_calibrate_898329_2.stderr parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898329/9a25d39c-9128-4551-b478-cd5d38257f0d_characterizeImage_898329_2.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898329/9a25d39c-9128-4551-b478-cd5d38257f0d_characterizeImage_898329_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 234 try 1 failed parsl.dataflow.dflow ERROR: Task 234 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 9847deae22a0da9db67da6a661dd1026 with result from task 234 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 235 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 235 try 0 failed parsl.dataflow.dflow INFO: Task 235 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898309/da51a41c-abeb-4ec0-be22-2f447f558076_calibrate_898309_2.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898309/da51a41c-abeb-4ec0-be22-2f447f558076_calibrate_898309_2.stderr parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898309/ab2a58da-d00a-4358-b4f7-75fcdce6e2e5_characterizeImage_898309_2.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898309/ab2a58da-d00a-4358-b4f7-75fcdce6e2e5_characterizeImage_898309_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 186 try 1 failed parsl.dataflow.dflow ERROR: Task 186 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry e6bf795532176e2d27dad486567f1e64 with result from task 186 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 187 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 187 try 0 failed parsl.dataflow.dflow INFO: Task 187 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898331/6b9caede-67c9-4866-8ebc-fdb91634f1a6_calibrate_898331_2.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898331/6b9caede-67c9-4866-8ebc-fdb91634f1a6_calibrate_898331_2.stderr parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898331/c0c18cff-9b6d-4db0-9644-84341b2c3bbd_characterizeImage_898331_2.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898331/c0c18cff-9b6d-4db0-9644-84341b2c3bbd_characterizeImage_898331_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 200 try 1 failed parsl.dataflow.dflow ERROR: Task 200 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 98a6b5e9eb625aa688a4be5b26f90646 with result from task 200 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 201 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 201 try 0 failed parsl.dataflow.dflow INFO: Task 201 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898347/38ca5688-8e80-40bd-acb7-c072c678893a_calibrate_898347_2.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898347/38ca5688-8e80-40bd-acb7-c072c678893a_calibrate_898347_2.stderr parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898347/cc7819d0-5467-422c-b477-88c64cf6b601_characterizeImage_898347_2.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898347/cc7819d0-5467-422c-b477-88c64cf6b601_characterizeImage_898347_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 192 try 1 failed parsl.dataflow.dflow ERROR: Task 192 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 20726f9423ee27a4b04345ffe73518b7 with result from task 192 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 193 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 193 try 0 failed parsl.dataflow.dflow INFO: Task 193 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898338/e8c0dd35-7612-41fc-ad55-eba14f74bcaf_calibrate_898338_2.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898338/e8c0dd35-7612-41fc-ad55-eba14f74bcaf_calibrate_898338_2.stderr parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898338/cb25c482-36e9-4bee-a408-37e1611db7cf_characterizeImage_898338_2.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898338/cb25c482-36e9-4bee-a408-37e1611db7cf_characterizeImage_898338_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:47:53 proc_lsst.multi:146 [INFO] found job 17008903 in provider astro proc_lsst.multi INFO: found job 17008903 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 32 try 1 failed parsl.dataflow.dflow ERROR: Task 32 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry ad83b57306beb23bf2701b4ec8428fe5 with result from task 32 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898351/f068b0a2-eb89-4a63-b361-e8d1e72e53c7_calibrate_898351_31.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898351/f068b0a2-eb89-4a63-b361-e8d1e72e53c7_calibrate_898351_31.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 216 try 1 failed parsl.dataflow.dflow ERROR: Task 216 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry f714afe43fbdf6119189762e695b5bfe with result from task 216 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 217 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 217 try 0 failed parsl.dataflow.dflow INFO: Task 217 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898363/55f82262-e8cb-4232-bb6c-c6a34f93c665_calibrate_898363_2.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898363/55f82262-e8cb-4232-bb6c-c6a34f93c665_calibrate_898363_2.stderr parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898363/15ceb904-a4e7-4b48-ac94-ffecf8eeed6b_characterizeImage_898363_2.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898363/15ceb904-a4e7-4b48-ac94-ffecf8eeed6b_characterizeImage_898363_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 228 try 1 failed parsl.dataflow.dflow ERROR: Task 228 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 5c901b5f51cce774e36848a802b4353c with result from task 228 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 229 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 229 try 0 failed parsl.dataflow.dflow INFO: Task 229 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898343/1dee1184-48f4-4663-8374-9b48b74bbe6f_calibrate_898343_2.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898343/1dee1184-48f4-4663-8374-9b48b74bbe6f_calibrate_898343_2.stderr parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898343/af00f831-886a-4c91-aed5-e90dc3c54710_characterizeImage_898343_2.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898343/af00f831-886a-4c91-aed5-e90dc3c54710_characterizeImage_898343_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:47:54 proc_lsst.multi:146 [INFO] found job 17008904 in provider astro proc_lsst.multi INFO: found job 17008904 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 240 try 1 failed parsl.dataflow.dflow ERROR: Task 240 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 0ce1d3fee239f4d14df437cdacf4f103 with result from task 240 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 241 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 241 try 0 failed parsl.dataflow.dflow INFO: Task 241 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898361/37e25c34-6e45-46ee-aa99-27cedb0708e9_calibrate_898361_2.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898361/37e25c34-6e45-46ee-aa99-27cedb0708e9_calibrate_898361_2.stderr parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898361/70cb0c22-8082-4558-a092-2d2526583506_characterizeImage_898361_2.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898361/70cb0c22-8082-4558-a092-2d2526583506_characterizeImage_898361_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 8 try 1 failed parsl.dataflow.dflow ERROR: Task 8 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 0ea846d7f41dde9c5463dfa6179c7af3 with result from task 8 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898366/3faa2151-682e-4b26-b2ce-a961451dc732_calibrate_898366_61.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898366/3faa2151-682e-4b26-b2ce-a961451dc732_calibrate_898366_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:47:55 proc_lsst.multi:146 [INFO] found job 17008905 in provider astro proc_lsst.multi INFO: found job 17008905 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 242 try 1 failed parsl.dataflow.dflow ERROR: Task 242 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 4074b8db1a55d97a7306e221d12c8572 with result from task 242 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 243 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 243 try 0 failed parsl.dataflow.dflow INFO: Task 243 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898314/18357552-2414-4948-ad99-31c044cd079b_calibrate_898314_2.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898314/18357552-2414-4948-ad99-31c044cd079b_calibrate_898314_2.stderr parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898314/0c096c3b-1ae9-4535-9515-ed489ad83844_characterizeImage_898314_2.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898314/0c096c3b-1ae9-4535-9515-ed489ad83844_characterizeImage_898314_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 236 try 1 failed parsl.dataflow.dflow ERROR: Task 236 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 6f9c150e9df8027239798bcf46aa33f0 with result from task 236 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 237 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 237 try 0 failed parsl.dataflow.dflow INFO: Task 237 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898298/bdf54913-95d6-4dd2-b2ec-304c760122d0_calibrate_898298_2.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898298/bdf54913-95d6-4dd2-b2ec-304c760122d0_calibrate_898298_2.stderr parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898298/a0281b0b-1e39-406c-851a-d3046c548e4c_characterizeImage_898298_2.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898298/a0281b0b-1e39-406c-851a-d3046c548e4c_characterizeImage_898298_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:47:57 proc_lsst.multi:146 [INFO] found job 17008906 in provider astro proc_lsst.multi INFO: found job 17008906 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 238 try 1 failed parsl.dataflow.dflow ERROR: Task 238 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 23c3f8d39c7bb146f77fc858d2f88f77 with result from task 238 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 239 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 239 try 0 failed parsl.dataflow.dflow INFO: Task 239 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898296/fee7e449-cd14-44db-85a6-5211c39ee5cd_calibrate_898296_2.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898296/fee7e449-cd14-44db-85a6-5211c39ee5cd_calibrate_898296_2.stderr parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898296/a19de25d-9b4d-4608-b993-e76d6f64d771_characterizeImage_898296_2.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898296/a19de25d-9b4d-4608-b993-e76d6f64d771_characterizeImage_898296_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 244 try 1 failed parsl.dataflow.dflow ERROR: Task 244 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 157cf90d46774aee3d879c243e268d49 with result from task 244 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 245 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 245 try 0 failed parsl.dataflow.dflow INFO: Task 245 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898377/b233a29c-5646-4901-ab27-177e7b477d49_calibrate_898377_2.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898377/b233a29c-5646-4901-ab27-177e7b477d49_calibrate_898377_2.stderr parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898377/73990b84-2916-4ebb-b290-1fbd20e627f9_characterizeImage_898377_2.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898377/73990b84-2916-4ebb-b290-1fbd20e627f9_characterizeImage_898377_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 220 try 1 failed parsl.dataflow.dflow ERROR: Task 220 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry a2683b89b666f946be5df544b264d391 with result from task 220 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 221 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 221 try 0 failed parsl.dataflow.dflow INFO: Task 221 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898335/e6bea804-7598-4c87-b818-7595719cf836_calibrate_898335_2.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898335/e6bea804-7598-4c87-b818-7595719cf836_calibrate_898335_2.stderr parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898335/281503df-63c1-4918-888e-a90e0e4d90ea_characterizeImage_898335_2.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898335/281503df-63c1-4918-888e-a90e0e4d90ea_characterizeImage_898335_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 246 try 1 failed parsl.dataflow.dflow ERROR: Task 246 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 210aca5b3c6c55476d15778b06af4561 with result from task 246 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 247 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 247 try 0 failed parsl.dataflow.dflow INFO: Task 247 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898357/21d6b1dd-cbba-4c25-a109-29e25e2ca0e8_calibrate_898357_2.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898357/21d6b1dd-cbba-4c25-a109-29e25e2ca0e8_calibrate_898357_2.stderr parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898357/5a7bd7a2-a9a9-41a8-9bb6-6917a13f6a9b_characterizeImage_898357_2.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898357/5a7bd7a2-a9a9-41a8-9bb6-6917a13f6a9b_characterizeImage_898357_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:47:58 proc_lsst.multi:146 [INFO] found job 17008907 in provider astro proc_lsst.multi INFO: found job 17008907 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 232 try 1 failed parsl.dataflow.dflow ERROR: Task 232 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 88c19e98bc647589cf501c6eef68b0f9 with result from task 232 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 233 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 233 try 0 failed parsl.dataflow.dflow INFO: Task 233 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898348/32d0f6b9-46e2-4a47-8176-d33db7b218c8_calibrate_898348_2.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898348/32d0f6b9-46e2-4a47-8176-d33db7b218c8_calibrate_898348_2.stderr parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898348/8791bf45-0123-47d0-97f9-3f528018f72b_characterizeImage_898348_2.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898348/8791bf45-0123-47d0-97f9-3f528018f72b_characterizeImage_898348_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:47:59 proc_lsst.multi:146 [INFO] found job 17008908 in provider astro proc_lsst.multi INFO: found job 17008908 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:00 proc_lsst.multi:146 [INFO] found job 17008909 in provider astro proc_lsst.multi INFO: found job 17008909 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 248 try 1 failed parsl.dataflow.dflow ERROR: Task 248 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 66068bc7a08514af0d0332311219ddb2 with result from task 248 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 249 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 249 try 0 failed parsl.dataflow.dflow INFO: Task 249 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898364/6d10303e-9ea3-48cb-898d-96d3ec56669a_calibrate_898364_2.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898364/6d10303e-9ea3-48cb-898d-96d3ec56669a_calibrate_898364_2.stderr parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898364/54421108-f5fa-4a91-9cb8-668488e2f15d_characterizeImage_898364_2.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/characterizeImage/898364/54421108-f5fa-4a91-9cb8-668488e2f15d_characterizeImage_898364_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:02 proc_lsst.multi:146 [INFO] found job 17008910 in provider astro proc_lsst.multi INFO: found job 17008910 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:03 proc_lsst.multi:146 [INFO] found job 17008911 in provider astro proc_lsst.multi INFO: found job 17008911 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:05 proc_lsst.multi:146 [INFO] found job 17008913 in provider astro proc_lsst.multi INFO: found job 17008913 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 55 try 1 failed parsl.dataflow.dflow ERROR: Task 55 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry e093fa6e2ddd0c7d3c73293fa8d0a1cf with result from task 55 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898331/9940692a-d823-4c18-bb68-1fb4b229ad0f_calibrate_898331_31.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898331/9940692a-d823-4c18-bb68-1fb4b229ad0f_calibrate_898331_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:06 proc_lsst.multi:146 [INFO] found job 17008914 in provider astro proc_lsst.multi INFO: found job 17008914 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:07 proc_lsst.multi:146 [INFO] found job 17008915 in provider astro proc_lsst.multi INFO: found job 17008915 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 14 try 1 failed parsl.dataflow.dflow ERROR: Task 14 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 976b0c75f578f168831d93d747f77b70 with result from task 14 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898356/1070cfd5-8ce6-41fc-a039-2d4ac71300c0_calibrate_898356_61.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898356/1070cfd5-8ce6-41fc-a039-2d4ac71300c0_calibrate_898356_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:08 proc_lsst.multi:146 [INFO] found job 17008917 in provider astro proc_lsst.multi INFO: found job 17008917 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:10 proc_lsst.multi:146 [INFO] found job 17008918 in provider astro proc_lsst.multi INFO: found job 17008918 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 25 try 1 failed parsl.dataflow.dflow ERROR: Task 25 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 24efdbd9c833bcea3669feca1f99e441 with result from task 25 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898319/61241879-2ac8-44d1-9098-dca5bb7dd64e_calibrate_898319_61.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898319/61241879-2ac8-44d1-9098-dca5bb7dd64e_calibrate_898319_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:11 proc_lsst.multi:146 [INFO] found job 17008919 in provider astro proc_lsst.multi INFO: found job 17008919 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 40 try 1 failed parsl.dataflow.dflow ERROR: Task 40 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry c7f9ef97f36aad0c3ad3f136aee93c45 with result from task 40 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898334/39685529-733b-419f-b862-f7a086c8ce6a_calibrate_898334_36.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898334/39685529-733b-419f-b862-f7a086c8ce6a_calibrate_898334_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:12 proc_lsst.multi:146 [INFO] found job 17008921 in provider astro proc_lsst.multi INFO: found job 17008921 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:13 proc_lsst.multi:146 [INFO] found job 17008922 in provider astro proc_lsst.multi INFO: found job 17008922 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 4 try 1 failed parsl.dataflow.dflow ERROR: Task 4 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 1b63349bb77c9e57be695c546ea22330 with result from task 4 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898375/21d6a845-9342-4d4b-907e-e59a7da8b7d0_calibrate_898375_61.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898375/21d6a845-9342-4d4b-907e-e59a7da8b7d0_calibrate_898375_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:15 proc_lsst.multi:146 [INFO] found job 17008923 in provider astro proc_lsst.multi INFO: found job 17008923 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:16 proc_lsst.multi:146 [INFO] found job 17008924 in provider astro proc_lsst.multi INFO: found job 17008924 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:17 proc_lsst.multi:146 [INFO] found job 17008925 in provider astro proc_lsst.multi INFO: found job 17008925 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:19 proc_lsst.multi:146 [INFO] found job 17008926 in provider astro proc_lsst.multi INFO: found job 17008926 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 23 try 1 failed parsl.dataflow.dflow ERROR: Task 23 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 8144802c995d4528edd2552745882d99 with result from task 23 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898345/5f136065-162d-49e3-9de7-31bf464a799c_calibrate_898345_61.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898345/5f136065-162d-49e3-9de7-31bf464a799c_calibrate_898345_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:20 proc_lsst.multi:146 [INFO] found job 17008927 in provider astro proc_lsst.multi INFO: found job 17008927 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 18 try 1 failed parsl.dataflow.dflow ERROR: Task 18 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 7b65f5e5c021803b62472089700ad251 with result from task 18 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898362/d41fc6e8-b2b1-467f-a720-63f17e62b88c_calibrate_898362_61.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898362/d41fc6e8-b2b1-467f-a720-63f17e62b88c_calibrate_898362_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:21 proc_lsst.multi:146 [INFO] found job 17008930 in provider astro proc_lsst.multi INFO: found job 17008930 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:22 proc_lsst.multi:146 [INFO] found job 17008934 in provider astro proc_lsst.multi INFO: found job 17008934 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:24 proc_lsst.multi:146 [INFO] found job 17008943 in provider escience_rtx6k proc_lsst.multi INFO: found job 17008943 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008943,17008949,17008950,17008951,17008952' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 10 try 1 failed parsl.dataflow.dflow ERROR: Task 10 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry b5cabc075ea894567234827bf4b39aaa with result from task 10 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898336/39bafb1f-6aae-4e48-9cf8-9bfc2976a642_calibrate_898336_61.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898336/39bafb1f-6aae-4e48-9cf8-9bfc2976a642_calibrate_898336_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008950 R 17008951 R 17008952 R 17008949 R 17008943 R parsl.providers.slurm.slurm DEBUG: Updating job 17008950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008943 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:25 proc_lsst.multi:146 [INFO] found job 17008949 in provider escience_rtx6k proc_lsst.multi INFO: found job 17008949 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008943,17008949,17008950,17008951,17008952' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 2 try 1 failed parsl.dataflow.dflow ERROR: Task 2 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 26209eddfa42689388c6f4e0bfdc4f38 with result from task 2 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898333/6df343bb-eaa6-4a48-a6cd-e1dd2bbc6bb7_calibrate_898333_61.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898333/6df343bb-eaa6-4a48-a6cd-e1dd2bbc6bb7_calibrate_898333_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008950 R 17008951 R 17008952 R 17008949 R 17008943 R parsl.providers.slurm.slurm DEBUG: Updating job 17008950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008943 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:26 proc_lsst.multi:146 [INFO] found job 17008950 in provider escience_rtx6k proc_lsst.multi INFO: found job 17008950 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008943,17008949,17008950,17008951,17008952' parsl.providers.slurm.slurm DEBUG: squeue returned 17008950 R 17008951 R 17008952 R 17008949 R 17008943 R parsl.providers.slurm.slurm DEBUG: Updating job 17008950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008943 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:27 proc_lsst.multi:146 [INFO] found job 17008951 in provider escience_rtx6k proc_lsst.multi INFO: found job 17008951 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008943,17008949,17008950,17008951,17008952' parsl.providers.slurm.slurm DEBUG: squeue returned 17008950 R 17008951 R 17008952 R 17008949 R 17008943 R parsl.providers.slurm.slurm DEBUG: Updating job 17008950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008943 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:28 proc_lsst.multi:146 [INFO] found job 17008952 in provider escience_rtx6k proc_lsst.multi INFO: found job 17008952 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008943,17008949,17008950,17008951,17008952' parsl.providers.slurm.slurm DEBUG: squeue returned 17008950 R 17008951 R 17008952 R 17008949 R 17008943 R parsl.providers.slurm.slurm DEBUG: Updating job 17008950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008943 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:29 proc_lsst.multi:146 [INFO] found job 17008953 in provider escience_a40 proc_lsst.multi INFO: found job 17008953 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008953,17008954,17008961' parsl.providers.slurm.slurm DEBUG: squeue returned 17008961 R 17008954 R 17008953 R parsl.providers.slurm.slurm DEBUG: Updating job 17008961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008953 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:30 proc_lsst.multi:146 [INFO] found job 17008954 in provider escience_a40 proc_lsst.multi INFO: found job 17008954 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008953,17008954,17008961' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 6 try 1 failed parsl.dataflow.dflow ERROR: Task 6 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry d97d0a7bdaf10f72d049ce95d7d29033 with result from task 6 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898374/0eeb0a0c-dc47-435f-bb6a-652c42051f5c_calibrate_898374_61.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898374/0eeb0a0c-dc47-435f-bb6a-652c42051f5c_calibrate_898374_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008961 R 17008954 R 17008953 R parsl.providers.slurm.slurm DEBUG: Updating job 17008961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008953 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:31 proc_lsst.multi:146 [INFO] found job 17008961 in provider escience_a40 proc_lsst.multi INFO: found job 17008961 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008953,17008954,17008961' parsl.providers.slurm.slurm DEBUG: squeue returned 17008961 R 17008954 R 17008953 R parsl.providers.slurm.slurm DEBUG: Updating job 17008961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008953 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 39, active_tasks = 20 parsl.dataflow.strategy DEBUG: Executor multi has 20 active tasks, 39/0 running/pending blocks, and 40 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22761558686992 2024-03-13 08:48:31 proc_lsst.multi:146 [INFO] found job 34092 in provider local proc_lsst.multi INFO: found job 34092 in provider local 2024-03-13 08:48:31 proc_lsst.multi:146 [INFO] found job 17008896 in provider astro proc_lsst.multi INFO: found job 17008896 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:32 proc_lsst.multi:146 [INFO] found job 17008898 in provider astro proc_lsst.multi INFO: found job 17008898 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:32 proc_lsst.multi:146 [INFO] found job 17008899 in provider astro proc_lsst.multi INFO: found job 17008899 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:32 proc_lsst.multi:146 [INFO] found job 17008900 in provider astro proc_lsst.multi INFO: found job 17008900 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:33 proc_lsst.multi:146 [INFO] found job 17008901 in provider astro proc_lsst.multi INFO: found job 17008901 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:33 proc_lsst.multi:146 [INFO] found job 17008902 in provider astro proc_lsst.multi INFO: found job 17008902 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:33 proc_lsst.multi:146 [INFO] found job 17008903 in provider astro proc_lsst.multi INFO: found job 17008903 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:34 proc_lsst.multi:146 [INFO] found job 17008904 in provider astro proc_lsst.multi INFO: found job 17008904 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:34 proc_lsst.multi:146 [INFO] found job 17008905 in provider astro proc_lsst.multi INFO: found job 17008905 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:34 proc_lsst.multi:146 [INFO] found job 17008906 in provider astro proc_lsst.multi INFO: found job 17008906 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 16 try 1 failed parsl.dataflow.dflow ERROR: Task 16 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 4b3acee8a57858b7460670ddb218b574 with result from task 16 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898363/9a3d4b9b-637b-4d54-9756-08db927791ea_calibrate_898363_61.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898363/9a3d4b9b-637b-4d54-9756-08db927791ea_calibrate_898363_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:35 proc_lsst.multi:146 [INFO] found job 17008907 in provider astro proc_lsst.multi INFO: found job 17008907 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:35 proc_lsst.multi:146 [INFO] found job 17008908 in provider astro proc_lsst.multi INFO: found job 17008908 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:35 proc_lsst.multi:146 [INFO] found job 17008909 in provider astro proc_lsst.multi INFO: found job 17008909 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:36 proc_lsst.multi:146 [INFO] found job 17008910 in provider astro proc_lsst.multi INFO: found job 17008910 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:36 proc_lsst.multi:146 [INFO] found job 17008911 in provider astro proc_lsst.multi INFO: found job 17008911 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:37 proc_lsst.multi:146 [INFO] found job 17008913 in provider astro proc_lsst.multi INFO: found job 17008913 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:37 proc_lsst.multi:146 [INFO] found job 17008914 in provider astro proc_lsst.multi INFO: found job 17008914 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:37 proc_lsst.multi:146 [INFO] found job 17008915 in provider astro proc_lsst.multi INFO: found job 17008915 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 48 try 1 failed parsl.dataflow.dflow ERROR: Task 48 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 81651c4a5cc7c31adf702f2a62e10b61 with result from task 48 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898382/26d7638c-432a-48eb-8a5e-2bd420831861_calibrate_898382_61.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898382/26d7638c-432a-48eb-8a5e-2bd420831861_calibrate_898382_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:38 proc_lsst.multi:146 [INFO] found job 17008917 in provider astro proc_lsst.multi INFO: found job 17008917 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:38 proc_lsst.multi:146 [INFO] found job 17008918 in provider astro proc_lsst.multi INFO: found job 17008918 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:38 proc_lsst.multi:146 [INFO] found job 17008919 in provider astro proc_lsst.multi INFO: found job 17008919 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:39 proc_lsst.multi:146 [INFO] found job 17008921 in provider astro proc_lsst.multi INFO: found job 17008921 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:39 proc_lsst.multi:146 [INFO] found job 17008922 in provider astro proc_lsst.multi INFO: found job 17008922 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:40 proc_lsst.multi:146 [INFO] found job 17008923 in provider astro proc_lsst.multi INFO: found job 17008923 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:40 proc_lsst.multi:146 [INFO] found job 17008924 in provider astro proc_lsst.multi INFO: found job 17008924 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:40 proc_lsst.multi:146 [INFO] found job 17008925 in provider astro proc_lsst.multi INFO: found job 17008925 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:41 proc_lsst.multi:146 [INFO] found job 17008926 in provider astro proc_lsst.multi INFO: found job 17008926 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:41 proc_lsst.multi:146 [INFO] found job 17008927 in provider astro proc_lsst.multi INFO: found job 17008927 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:41 proc_lsst.multi:146 [INFO] found job 17008930 in provider astro proc_lsst.multi INFO: found job 17008930 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:42 proc_lsst.multi:146 [INFO] found job 17008934 in provider astro proc_lsst.multi INFO: found job 17008934 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:42 proc_lsst.multi:146 [INFO] found job 17008943 in provider escience_rtx6k proc_lsst.multi INFO: found job 17008943 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008943,17008949,17008950,17008951,17008952' parsl.providers.slurm.slurm DEBUG: squeue returned 17008950 R 17008951 R 17008952 R 17008949 R 17008943 R parsl.providers.slurm.slurm DEBUG: Updating job 17008950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008943 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:42 proc_lsst.multi:146 [INFO] found job 17008949 in provider escience_rtx6k proc_lsst.multi INFO: found job 17008949 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008943,17008949,17008950,17008951,17008952' parsl.providers.slurm.slurm DEBUG: squeue returned 17008950 R 17008951 R 17008952 R 17008949 R 17008943 R parsl.providers.slurm.slurm DEBUG: Updating job 17008950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008943 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:43 proc_lsst.multi:146 [INFO] found job 17008950 in provider escience_rtx6k proc_lsst.multi INFO: found job 17008950 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008943,17008949,17008950,17008951,17008952' parsl.providers.slurm.slurm DEBUG: squeue returned 17008950 R 17008951 R 17008952 R 17008949 R 17008943 R parsl.providers.slurm.slurm DEBUG: Updating job 17008950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008943 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:43 proc_lsst.multi:146 [INFO] found job 17008951 in provider escience_rtx6k proc_lsst.multi INFO: found job 17008951 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008943,17008949,17008950,17008951,17008952' parsl.providers.slurm.slurm DEBUG: squeue returned 17008950 R 17008951 R 17008952 R 17008949 R 17008943 R parsl.providers.slurm.slurm DEBUG: Updating job 17008950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008943 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:44 proc_lsst.multi:146 [INFO] found job 17008952 in provider escience_rtx6k proc_lsst.multi INFO: found job 17008952 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008943,17008949,17008950,17008951,17008952' parsl.providers.slurm.slurm DEBUG: squeue returned 17008950 R 17008951 R 17008952 R 17008949 R 17008943 R parsl.providers.slurm.slurm DEBUG: Updating job 17008950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008943 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:44 proc_lsst.multi:146 [INFO] found job 17008953 in provider escience_a40 proc_lsst.multi INFO: found job 17008953 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008953,17008954,17008961' parsl.providers.slurm.slurm DEBUG: squeue returned 17008961 R 17008954 R 17008953 R parsl.providers.slurm.slurm DEBUG: Updating job 17008961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008953 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:44 proc_lsst.multi:146 [INFO] found job 17008954 in provider escience_a40 proc_lsst.multi INFO: found job 17008954 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008953,17008954,17008961' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 46 try 1 failed parsl.dataflow.dflow ERROR: Task 46 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 8f8e36e3614cf9d2377fccbbc80d45b4 with result from task 46 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898332/1afe740a-c14b-4016-ba53-e7fbd2c9f2fc_calibrate_898332_61.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898332/1afe740a-c14b-4016-ba53-e7fbd2c9f2fc_calibrate_898332_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008961 R 17008954 R 17008953 R parsl.providers.slurm.slurm DEBUG: Updating job 17008961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008953 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:48:45 proc_lsst.multi:146 [INFO] found job 17008961 in provider escience_a40 proc_lsst.multi INFO: found job 17008961 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008953,17008954,17008961' parsl.providers.slurm.slurm DEBUG: squeue returned 17008961 R 17008954 R 17008953 R parsl.providers.slurm.slurm DEBUG: Updating job 17008961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008953 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 39, active_tasks = 17 parsl.dataflow.strategy DEBUG: Executor multi has 17 active tasks, 39/0 running/pending blocks, and 40 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22761558686992 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 39, active_tasks = 17 parsl.dataflow.strategy DEBUG: Executor multi has 17 active tasks, 39/0 running/pending blocks, and 40 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22761558686992 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 45 try 1 failed parsl.dataflow.dflow ERROR: Task 45 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 98fd1d3c7031e1dbd8391f88c1a617db with result from task 45 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898322/917bf878-02ae-462d-8477-3109632be425_calibrate_898322_61.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898322/917bf878-02ae-462d-8477-3109632be425_calibrate_898322_61.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 47 try 1 failed parsl.dataflow.dflow ERROR: Task 47 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 495796d6dd275cdebea5a7947f2db62b with result from task 47 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898365/68f3688a-de27-4713-83e4-4a8332c2e510_calibrate_898365_61.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898365/68f3688a-de27-4713-83e4-4a8332c2e510_calibrate_898365_61.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 19 try 1 failed parsl.dataflow.dflow ERROR: Task 19 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 2dd5cc0de2690a25ddf96527927bdc84 with result from task 19 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898335/9ee28e39-dd1f-40cc-91e9-458e31ad9e35_calibrate_898335_61.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898335/9ee28e39-dd1f-40cc-91e9-458e31ad9e35_calibrate_898335_61.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 30 try 1 failed parsl.dataflow.dflow ERROR: Task 30 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 6fd2140ec551b4a45f963c0c020e2ebf with result from task 30 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898334/b995a848-ee4d-4405-a8c0-dda0691b96b0_calibrate_898334_61.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898334/b995a848-ee4d-4405-a8c0-dda0691b96b0_calibrate_898334_61.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 39, active_tasks = 13 parsl.dataflow.strategy DEBUG: Executor multi has 13 active tasks, 39/0 running/pending blocks, and 40 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22761558686992 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 31 try 1 failed parsl.dataflow.dflow ERROR: Task 31 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry bcfe0a168ab99bf456c8b1f0cdfa7d07 with result from task 31 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898343/64d0e1db-ccf1-458e-bbd9-d102c90d7e8b_calibrate_898343_61.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898343/64d0e1db-ccf1-458e-bbd9-d102c90d7e8b_calibrate_898343_61.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 53 try 1 failed parsl.dataflow.dflow ERROR: Task 53 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 5643377b5f14ba72e85f5850cb6406a2 with result from task 53 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898346/148e6dc2-a926-488e-8cfb-2a923573e179_calibrate_898346_61.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898346/148e6dc2-a926-488e-8cfb-2a923573e179_calibrate_898346_61.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 39, active_tasks = 11 parsl.dataflow.strategy DEBUG: Executor multi has 11 active tasks, 39/0 running/pending blocks, and 40 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22761558686992 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 11 try 1 failed parsl.dataflow.dflow ERROR: Task 11 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry a02de1cca403ca71bfd1ce8526d5b68a with result from task 11 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898342/cf783a12-8612-47c1-ad91-69a33e09a85c_calibrate_898342_61.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898342/cf783a12-8612-47c1-ad91-69a33e09a85c_calibrate_898342_61.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 51 try 1 failed parsl.dataflow.dflow ERROR: Task 51 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 0a6b63a644cafb3f45916f28e2dcab00 with result from task 51 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898353/b15df392-ec28-447e-81c3-72d987d085c6_calibrate_898353_61.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898353/b15df392-ec28-447e-81c3-72d987d085c6_calibrate_898353_61.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 39, active_tasks = 9 parsl.dataflow.strategy DEBUG: Executor multi has 9 active tasks, 39/0 running/pending blocks, and 40 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22761558686992 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 33 try 1 failed parsl.dataflow.dflow ERROR: Task 33 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry c88a06f6a83e0593b438fb1774dd2460 with result from task 33 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898344/bf5bb1bb-5ab6-47f9-8e07-57808bd6e728_calibrate_898344_61.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898344/bf5bb1bb-5ab6-47f9-8e07-57808bd6e728_calibrate_898344_61.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 50 try 1 failed parsl.dataflow.dflow ERROR: Task 50 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 362734b5a790c00cc0b62964f63e77c9 with result from task 50 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898323/93fbc6ff-c819-4d23-b892-475cc0988494_calibrate_898323_61.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898323/93fbc6ff-c819-4d23-b892-475cc0988494_calibrate_898323_61.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 41 try 1 failed parsl.dataflow.dflow ERROR: Task 41 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 9ca2ae210d2c757b4188bbf1deab3a05 with result from task 41 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898324/8d56e6fa-7a2c-4b9b-afa3-e24d76c7887b_calibrate_898324_61.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898324/8d56e6fa-7a2c-4b9b-afa3-e24d76c7887b_calibrate_898324_61.stderr 2024-03-13 08:49:05 proc_lsst.multi:146 [INFO] found job 34092 in provider local proc_lsst.multi INFO: found job 34092 in provider local 2024-03-13 08:49:05 proc_lsst.multi:146 [INFO] found job 17008896 in provider astro proc_lsst.multi INFO: found job 17008896 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:49:05 proc_lsst.multi:146 [INFO] found job 17008898 in provider astro proc_lsst.multi INFO: found job 17008898 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:49:06 proc_lsst.multi:146 [INFO] found job 17008899 in provider astro proc_lsst.multi INFO: found job 17008899 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:49:06 proc_lsst.multi:146 [INFO] found job 17008900 in provider astro proc_lsst.multi INFO: found job 17008900 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:49:06 proc_lsst.multi:146 [INFO] found job 17008901 in provider astro proc_lsst.multi INFO: found job 17008901 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:49:06 proc_lsst.multi:146 [INFO] found job 17008902 in provider astro proc_lsst.multi INFO: found job 17008902 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:49:06 proc_lsst.multi:146 [INFO] found job 17008903 in provider astro proc_lsst.multi INFO: found job 17008903 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:49:07 proc_lsst.multi:146 [INFO] found job 17008904 in provider astro proc_lsst.multi INFO: found job 17008904 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:49:07 proc_lsst.multi:146 [INFO] found job 17008905 in provider astro proc_lsst.multi INFO: found job 17008905 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:49:07 proc_lsst.multi:146 [INFO] found job 17008906 in provider astro proc_lsst.multi INFO: found job 17008906 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:49:07 proc_lsst.multi:146 [INFO] found job 17008907 in provider astro proc_lsst.multi INFO: found job 17008907 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 24 try 1 failed parsl.dataflow.dflow ERROR: Task 24 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 4a0f9e7707a025e86a9bce98ffc50f66 with result from task 24 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898331/5c785658-adcd-419d-b339-11f13f7ccddd_calibrate_898331_61.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898331/5c785658-adcd-419d-b339-11f13f7ccddd_calibrate_898331_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:49:08 proc_lsst.multi:146 [INFO] found job 17008908 in provider astro proc_lsst.multi INFO: found job 17008908 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:49:08 proc_lsst.multi:146 [INFO] found job 17008909 in provider astro proc_lsst.multi INFO: found job 17008909 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:49:08 proc_lsst.multi:146 [INFO] found job 17008910 in provider astro proc_lsst.multi INFO: found job 17008910 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:49:08 proc_lsst.multi:146 [INFO] found job 17008911 in provider astro proc_lsst.multi INFO: found job 17008911 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 36 try 1 failed parsl.dataflow.dflow ERROR: Task 36 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 61aee1dbd6fba070a7190edb6496165b with result from task 36 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898309/82e1efa9-7097-42b4-aede-e23e05ddbd81_calibrate_898309_61.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898309/82e1efa9-7097-42b4-aede-e23e05ddbd81_calibrate_898309_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:49:08 proc_lsst.multi:146 [INFO] found job 17008913 in provider astro proc_lsst.multi INFO: found job 17008913 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:49:08 proc_lsst.multi:146 [INFO] found job 17008914 in provider astro proc_lsst.multi INFO: found job 17008914 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:49:09 proc_lsst.multi:146 [INFO] found job 17008915 in provider astro proc_lsst.multi INFO: found job 17008915 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:49:09 proc_lsst.multi:146 [INFO] found job 17008917 in provider astro proc_lsst.multi INFO: found job 17008917 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 17 try 1 failed parsl.dataflow.dflow ERROR: Task 17 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry c3be2f6c16e1ce750d1a80c216b86dbf with result from task 17 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898369/53ab6566-4b30-4473-a020-ede08ff4c708_calibrate_898369_61.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898369/53ab6566-4b30-4473-a020-ede08ff4c708_calibrate_898369_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:49:09 proc_lsst.multi:146 [INFO] found job 17008918 in provider astro proc_lsst.multi INFO: found job 17008918 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:49:09 proc_lsst.multi:146 [INFO] found job 17008919 in provider astro proc_lsst.multi INFO: found job 17008919 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:49:09 proc_lsst.multi:146 [INFO] found job 17008921 in provider astro proc_lsst.multi INFO: found job 17008921 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:49:09 proc_lsst.multi:146 [INFO] found job 17008922 in provider astro proc_lsst.multi INFO: found job 17008922 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:49:09 proc_lsst.multi:146 [INFO] found job 17008923 in provider astro proc_lsst.multi INFO: found job 17008923 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:49:10 proc_lsst.multi:146 [INFO] found job 17008924 in provider astro proc_lsst.multi INFO: found job 17008924 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:49:10 proc_lsst.multi:146 [INFO] found job 17008925 in provider astro proc_lsst.multi INFO: found job 17008925 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:49:10 proc_lsst.multi:146 [INFO] found job 17008926 in provider astro proc_lsst.multi INFO: found job 17008926 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:49:10 proc_lsst.multi:146 [INFO] found job 17008927 in provider astro proc_lsst.multi INFO: found job 17008927 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:49:10 proc_lsst.multi:146 [INFO] found job 17008930 in provider astro proc_lsst.multi INFO: found job 17008930 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:49:10 proc_lsst.multi:146 [INFO] found job 17008934 in provider astro proc_lsst.multi INFO: found job 17008934 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008896,17008898,17008899,17008900,17008901,17008902,17008903,17008904,17008905,17008906,17008907,17008908,17008909,17008910,17008911,17008913,17008914,17008915,17008917,17008918,17008919,17008921,17008922,17008923,17008924,17008925,17008926,17008927,17008930,17008934' parsl.providers.slurm.slurm DEBUG: squeue returned 17008934 R 17008930 R 17008925 R 17008926 R 17008927 R 17008923 R 17008924 R 17008921 R 17008922 R 17008917 R 17008918 R 17008919 R 17008909 R 17008910 R 17008911 R 17008913 R 17008914 R 17008915 R 17008904 R 17008905 R 17008906 R 17008907 R 17008908 R 17008898 R 17008899 R 17008900 R 17008901 R 17008902 R 17008903 R 17008896 R parsl.providers.slurm.slurm DEBUG: Updating job 17008934 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008930 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008925 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008926 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008927 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008923 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008924 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008921 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008922 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008917 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008918 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008919 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008913 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008914 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008915 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008896 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:49:10 proc_lsst.multi:146 [INFO] found job 17008943 in provider escience_rtx6k proc_lsst.multi INFO: found job 17008943 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008943,17008949,17008950,17008951,17008952' parsl.providers.slurm.slurm DEBUG: squeue returned 17008950 R 17008951 R 17008952 R 17008949 R 17008943 R parsl.providers.slurm.slurm DEBUG: Updating job 17008950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008943 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:49:10 proc_lsst.multi:146 [INFO] found job 17008949 in provider escience_rtx6k proc_lsst.multi INFO: found job 17008949 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008943,17008949,17008950,17008951,17008952' parsl.providers.slurm.slurm DEBUG: squeue returned 17008950 R 17008951 R 17008952 R 17008949 R 17008943 R parsl.providers.slurm.slurm DEBUG: Updating job 17008950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008943 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:49:10 proc_lsst.multi:146 [INFO] found job 17008950 in provider escience_rtx6k proc_lsst.multi INFO: found job 17008950 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008943,17008949,17008950,17008951,17008952' parsl.providers.slurm.slurm DEBUG: squeue returned 17008950 R 17008951 R 17008952 R 17008949 R 17008943 R parsl.providers.slurm.slurm DEBUG: Updating job 17008950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008943 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:49:11 proc_lsst.multi:146 [INFO] found job 17008951 in provider escience_rtx6k proc_lsst.multi INFO: found job 17008951 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008943,17008949,17008950,17008951,17008952' parsl.providers.slurm.slurm DEBUG: squeue returned 17008950 R 17008951 R 17008952 R 17008949 R 17008943 R parsl.providers.slurm.slurm DEBUG: Updating job 17008950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008943 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:49:11 proc_lsst.multi:146 [INFO] found job 17008952 in provider escience_rtx6k proc_lsst.multi INFO: found job 17008952 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008943,17008949,17008950,17008951,17008952' parsl.providers.slurm.slurm DEBUG: squeue returned 17008950 R 17008951 R 17008952 R 17008949 R 17008943 R parsl.providers.slurm.slurm DEBUG: Updating job 17008950 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008951 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008952 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008949 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008943 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:49:11 proc_lsst.multi:146 [INFO] found job 17008953 in provider escience_a40 proc_lsst.multi INFO: found job 17008953 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008953,17008954,17008961' parsl.providers.slurm.slurm DEBUG: squeue returned 17008961 R 17008954 R 17008953 R parsl.providers.slurm.slurm DEBUG: Updating job 17008961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008953 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:49:11 proc_lsst.multi:146 [INFO] found job 17008954 in provider escience_a40 proc_lsst.multi INFO: found job 17008954 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008953,17008954,17008961' parsl.providers.slurm.slurm DEBUG: squeue returned 17008961 R 17008954 R 17008953 R parsl.providers.slurm.slurm DEBUG: Updating job 17008961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008953 with slurm status R to parsl state JobState.RUNNING 2024-03-13 08:49:11 proc_lsst.multi:146 [INFO] found job 17008961 in provider escience_a40 proc_lsst.multi INFO: found job 17008961 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17008953,17008954,17008961' parsl.providers.slurm.slurm DEBUG: squeue returned 17008961 R 17008954 R 17008953 R parsl.providers.slurm.slurm DEBUG: Updating job 17008961 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008954 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17008953 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 39, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 39/0 running/pending blocks, and 40 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22761558686992 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 39, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 39/0 running/pending blocks, and 40 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22761558686992 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 39 try 1 failed parsl.dataflow.dflow ERROR: Task 39 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 5d36b7e58d4a9ec85cab8d9824bd8fee with result from task 39 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898341/22b89990-f444-407a-a373-03704d655796_calibrate_898341_61.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898341/22b89990-f444-407a-a373-03704d655796_calibrate_898341_61.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 37 try 1 failed parsl.dataflow.dflow ERROR: Task 37 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 9bbeccbae9becdd64a681642865bf01b with result from task 37 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898329/0ba2197b-c2e4-4473-a0f4-3c1dedb841c1_calibrate_898329_61.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898329/0ba2197b-c2e4-4473-a0f4-3c1dedb841c1_calibrate_898329_61.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 39, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 39/0 running/pending blocks, and 40 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22761558686992 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 38 try 1 failed parsl.dataflow.dflow ERROR: Task 38 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 2cd84899ae596971a5725f38d92779eb with result from task 38 parsl.dataflow.dflow INFO: DFK cleanup initiated parsl.dataflow.dflow INFO: Summary of tasks in DFK: parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0 parsl.dataflow.dflow INFO: Tasks in state States.pending: 0 parsl.dataflow.dflow INFO: Tasks in state States.running: 0 parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 0 parsl.dataflow.dflow INFO: Tasks in state States.failed: 153 parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 97 parsl.dataflow.dflow INFO: Tasks in state States.launched: 0 parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0 parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0 parsl.dataflow.dflow INFO: Tasks in state States.joining: 0 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898337/44b1f232-8e71-4f34-8980-bb9b465c63b1_calibrate_898337_61.stdout parsl.dataflow.dflow INFO: End of summary parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/submit/DEEP/20190926/B1a/science#step1/20240313T084347Z/logs/calibrate/898337/44b1f232-8e71-4f34-8980-bb9b465c63b1_calibrate_898337_61.stderr parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Closing job status poller parsl.dataflow.dflow INFO: Terminated job status poller parsl.dataflow.dflow INFO: Scaling in and shutting down executors parsl.dataflow.dflow INFO: Scaling in executor multi parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=39, block_ids=[] parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 39 blocks parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 798a2de91e9c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 798a2de91e9c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 35c829d2e10a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 35c829d2e10a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b403d15b199a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b403d15b199a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1260a3fe2480 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1260a3fe2480 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e519c9136050 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e519c9136050 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 25f40a64547c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 25f40a64547c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5732ca718c9d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5732ca718c9d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 00646f24352a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 00646f24352a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c2905ca3630d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c2905ca3630d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2e4e124f340d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2e4e124f340d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a14b403d4c6a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a14b403d4c6a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 00309afa922c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 00309afa922c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0c016b593e8d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0c016b593e8d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 01dad63fc876 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 01dad63fc876 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2d7ea9847b32 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2d7ea9847b32 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4cefed8efe2e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4cefed8efe2e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cbf3fe6281f5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cbf3fe6281f5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9be4b3680281 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9be4b3680281 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 728c55c969a5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 728c55c969a5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 56460a6b4b28 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 56460a6b4b28 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6b5fffe4cc2e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6b5fffe4cc2e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 539f39c3bdae parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 539f39c3bdae parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bbd1ca1d50b9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bbd1ca1d50b9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bbc04e5bf1e6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bbc04e5bf1e6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2404f15289e5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2404f15289e5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ebc299b1df43 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ebc299b1df43 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6ebb00836fd8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6ebb00836fd8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3d9f1289b0a4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3d9f1289b0a4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dce109052f6e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dce109052f6e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 17dbeb20a6a9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 17dbeb20a6a9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c67f9348e1d5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c67f9348e1d5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1efcb75ad15a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1efcb75ad15a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 455634f9812a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 455634f9812a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7678015c0967 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7678015c0967 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6603ab7e61b3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6603ab7e61b3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 74dcf6803ff1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 74dcf6803ff1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 313fb9e97657 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 313fb9e97657 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8a65de87c9d3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8a65de87c9d3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 79aed85a3205 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 79aed85a3205 2024-03-13 08:49:21 proc_lsst.multi:146 [INFO] found job 34092 in provider local proc_lsst.multi INFO: found job 34092 in provider local 2024-03-13 08:49:21 proc_lsst.multi:201 [INFO] cancelling 34092 on provider local proc_lsst.multi INFO: cancelling 34092 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 34092 2024-03-13 08:49:21 proc_lsst.multi:146 [INFO] found job 17008953 in provider escience_a40 proc_lsst.multi INFO: found job 17008953 in provider escience_a40 2024-03-13 08:49:21 proc_lsst.multi:201 [INFO] cancelling 17008953 on provider escience_a40 proc_lsst.multi INFO: cancelling 17008953 on provider escience_a40 2024-03-13 08:49:21 proc_lsst.multi:146 [INFO] found job 17008954 in provider escience_a40 proc_lsst.multi INFO: found job 17008954 in provider escience_a40 2024-03-13 08:49:21 proc_lsst.multi:201 [INFO] cancelling 17008954 on provider escience_a40 proc_lsst.multi INFO: cancelling 17008954 on provider escience_a40 2024-03-13 08:49:21 proc_lsst.multi:146 [INFO] found job 17008898 in provider astro proc_lsst.multi INFO: found job 17008898 in provider astro 2024-03-13 08:49:21 proc_lsst.multi:201 [INFO] cancelling 17008898 on provider astro proc_lsst.multi INFO: cancelling 17008898 on provider astro 2024-03-13 08:49:21 proc_lsst.multi:146 [INFO] found job 17008943 in provider escience_rtx6k proc_lsst.multi INFO: found job 17008943 in provider escience_rtx6k 2024-03-13 08:49:21 proc_lsst.multi:201 [INFO] cancelling 17008943 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17008943 on provider escience_rtx6k 2024-03-13 08:49:21 proc_lsst.multi:146 [INFO] found job 17008907 in provider astro proc_lsst.multi INFO: found job 17008907 in provider astro 2024-03-13 08:49:21 proc_lsst.multi:201 [INFO] cancelling 17008907 on provider astro proc_lsst.multi INFO: cancelling 17008907 on provider astro 2024-03-13 08:49:21 proc_lsst.multi:146 [INFO] found job 17008901 in provider astro proc_lsst.multi INFO: found job 17008901 in provider astro 2024-03-13 08:49:21 proc_lsst.multi:201 [INFO] cancelling 17008901 on provider astro proc_lsst.multi INFO: cancelling 17008901 on provider astro 2024-03-13 08:49:21 proc_lsst.multi:146 [INFO] found job 17008961 in provider escience_a40 proc_lsst.multi INFO: found job 17008961 in provider escience_a40 2024-03-13 08:49:21 proc_lsst.multi:201 [INFO] cancelling 17008961 on provider escience_a40 proc_lsst.multi INFO: cancelling 17008961 on provider escience_a40 2024-03-13 08:49:21 proc_lsst.multi:146 [INFO] found job 17008923 in provider astro proc_lsst.multi INFO: found job 17008923 in provider astro 2024-03-13 08:49:21 proc_lsst.multi:201 [INFO] cancelling 17008923 on provider astro proc_lsst.multi INFO: cancelling 17008923 on provider astro 2024-03-13 08:49:22 proc_lsst.multi:146 [INFO] found job 17008910 in provider astro proc_lsst.multi INFO: found job 17008910 in provider astro 2024-03-13 08:49:22 proc_lsst.multi:201 [INFO] cancelling 17008910 on provider astro proc_lsst.multi INFO: cancelling 17008910 on provider astro 2024-03-13 08:49:22 proc_lsst.multi:146 [INFO] found job 17008904 in provider astro proc_lsst.multi INFO: found job 17008904 in provider astro 2024-03-13 08:49:22 proc_lsst.multi:201 [INFO] cancelling 17008904 on provider astro proc_lsst.multi INFO: cancelling 17008904 on provider astro 2024-03-13 08:49:22 proc_lsst.multi:146 [INFO] found job 17008919 in provider astro proc_lsst.multi INFO: found job 17008919 in provider astro 2024-03-13 08:49:22 proc_lsst.multi:201 [INFO] cancelling 17008919 on provider astro proc_lsst.multi INFO: cancelling 17008919 on provider astro 2024-03-13 08:49:22 proc_lsst.multi:146 [INFO] found job 17008926 in provider astro proc_lsst.multi INFO: found job 17008926 in provider astro 2024-03-13 08:49:22 proc_lsst.multi:201 [INFO] cancelling 17008926 on provider astro proc_lsst.multi INFO: cancelling 17008926 on provider astro 2024-03-13 08:49:22 proc_lsst.multi:146 [INFO] found job 17008922 in provider astro proc_lsst.multi INFO: found job 17008922 in provider astro 2024-03-13 08:49:22 proc_lsst.multi:201 [INFO] cancelling 17008922 on provider astro proc_lsst.multi INFO: cancelling 17008922 on provider astro 2024-03-13 08:49:22 proc_lsst.multi:146 [INFO] found job 17008918 in provider astro proc_lsst.multi INFO: found job 17008918 in provider astro 2024-03-13 08:49:22 proc_lsst.multi:201 [INFO] cancelling 17008918 on provider astro proc_lsst.multi INFO: cancelling 17008918 on provider astro 2024-03-13 08:49:22 proc_lsst.multi:146 [INFO] found job 17008899 in provider astro proc_lsst.multi INFO: found job 17008899 in provider astro 2024-03-13 08:49:22 proc_lsst.multi:201 [INFO] cancelling 17008899 on provider astro proc_lsst.multi INFO: cancelling 17008899 on provider astro 2024-03-13 08:49:22 proc_lsst.multi:146 [INFO] found job 17008927 in provider astro proc_lsst.multi INFO: found job 17008927 in provider astro 2024-03-13 08:49:22 proc_lsst.multi:201 [INFO] cancelling 17008927 on provider astro proc_lsst.multi INFO: cancelling 17008927 on provider astro 2024-03-13 08:49:22 proc_lsst.multi:146 [INFO] found job 17008924 in provider astro proc_lsst.multi INFO: found job 17008924 in provider astro 2024-03-13 08:49:22 proc_lsst.multi:201 [INFO] cancelling 17008924 on provider astro proc_lsst.multi INFO: cancelling 17008924 on provider astro 2024-03-13 08:49:23 proc_lsst.multi:146 [INFO] found job 17008925 in provider astro proc_lsst.multi INFO: found job 17008925 in provider astro 2024-03-13 08:49:23 proc_lsst.multi:201 [INFO] cancelling 17008925 on provider astro proc_lsst.multi INFO: cancelling 17008925 on provider astro 2024-03-13 08:49:23 proc_lsst.multi:146 [INFO] found job 17008921 in provider astro proc_lsst.multi INFO: found job 17008921 in provider astro 2024-03-13 08:49:23 proc_lsst.multi:201 [INFO] cancelling 17008921 on provider astro proc_lsst.multi INFO: cancelling 17008921 on provider astro 2024-03-13 08:49:23 proc_lsst.multi:146 [INFO] found job 17008902 in provider astro proc_lsst.multi INFO: found job 17008902 in provider astro 2024-03-13 08:49:23 proc_lsst.multi:201 [INFO] cancelling 17008902 on provider astro proc_lsst.multi INFO: cancelling 17008902 on provider astro 2024-03-13 08:49:23 proc_lsst.multi:146 [INFO] found job 17008930 in provider astro proc_lsst.multi INFO: found job 17008930 in provider astro 2024-03-13 08:49:23 proc_lsst.multi:201 [INFO] cancelling 17008930 on provider astro proc_lsst.multi INFO: cancelling 17008930 on provider astro 2024-03-13 08:49:23 proc_lsst.multi:146 [INFO] found job 17008914 in provider astro proc_lsst.multi INFO: found job 17008914 in provider astro 2024-03-13 08:49:23 proc_lsst.multi:201 [INFO] cancelling 17008914 on provider astro proc_lsst.multi INFO: cancelling 17008914 on provider astro 2024-03-13 08:49:23 proc_lsst.multi:146 [INFO] found job 17008913 in provider astro proc_lsst.multi INFO: found job 17008913 in provider astro 2024-03-13 08:49:23 proc_lsst.multi:201 [INFO] cancelling 17008913 on provider astro proc_lsst.multi INFO: cancelling 17008913 on provider astro 2024-03-13 08:49:23 proc_lsst.multi:146 [INFO] found job 17008911 in provider astro proc_lsst.multi INFO: found job 17008911 in provider astro 2024-03-13 08:49:23 proc_lsst.multi:201 [INFO] cancelling 17008911 on provider astro proc_lsst.multi INFO: cancelling 17008911 on provider astro 2024-03-13 08:49:23 proc_lsst.multi:146 [INFO] found job 17008896 in provider astro proc_lsst.multi INFO: found job 17008896 in provider astro 2024-03-13 08:49:23 proc_lsst.multi:201 [INFO] cancelling 17008896 on provider astro proc_lsst.multi INFO: cancelling 17008896 on provider astro 2024-03-13 08:49:23 proc_lsst.multi:146 [INFO] found job 17008909 in provider astro proc_lsst.multi INFO: found job 17008909 in provider astro 2024-03-13 08:49:23 proc_lsst.multi:201 [INFO] cancelling 17008909 on provider astro proc_lsst.multi INFO: cancelling 17008909 on provider astro 2024-03-13 08:49:24 proc_lsst.multi:146 [INFO] found job 17008915 in provider astro proc_lsst.multi INFO: found job 17008915 in provider astro 2024-03-13 08:49:24 proc_lsst.multi:201 [INFO] cancelling 17008915 on provider astro proc_lsst.multi INFO: cancelling 17008915 on provider astro 2024-03-13 08:49:24 proc_lsst.multi:146 [INFO] found job 17008900 in provider astro proc_lsst.multi INFO: found job 17008900 in provider astro 2024-03-13 08:49:24 proc_lsst.multi:201 [INFO] cancelling 17008900 on provider astro proc_lsst.multi INFO: cancelling 17008900 on provider astro 2024-03-13 08:49:24 proc_lsst.multi:146 [INFO] found job 17008949 in provider escience_rtx6k proc_lsst.multi INFO: found job 17008949 in provider escience_rtx6k 2024-03-13 08:49:24 proc_lsst.multi:201 [INFO] cancelling 17008949 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17008949 on provider escience_rtx6k 2024-03-13 08:49:24 proc_lsst.multi:146 [INFO] found job 17008917 in provider astro proc_lsst.multi INFO: found job 17008917 in provider astro 2024-03-13 08:49:24 proc_lsst.multi:201 [INFO] cancelling 17008917 on provider astro proc_lsst.multi INFO: cancelling 17008917 on provider astro 2024-03-13 08:49:24 proc_lsst.multi:146 [INFO] found job 17008934 in provider astro proc_lsst.multi INFO: found job 17008934 in provider astro 2024-03-13 08:49:24 proc_lsst.multi:201 [INFO] cancelling 17008934 on provider astro proc_lsst.multi INFO: cancelling 17008934 on provider astro 2024-03-13 08:49:24 proc_lsst.multi:146 [INFO] found job 17008952 in provider escience_rtx6k proc_lsst.multi INFO: found job 17008952 in provider escience_rtx6k 2024-03-13 08:49:24 proc_lsst.multi:201 [INFO] cancelling 17008952 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17008952 on provider escience_rtx6k 2024-03-13 08:49:24 proc_lsst.multi:146 [INFO] found job 17008905 in provider astro proc_lsst.multi INFO: found job 17008905 in provider astro 2024-03-13 08:49:24 proc_lsst.multi:201 [INFO] cancelling 17008905 on provider astro proc_lsst.multi INFO: cancelling 17008905 on provider astro 2024-03-13 08:49:24 proc_lsst.multi:146 [INFO] found job 17008908 in provider astro proc_lsst.multi INFO: found job 17008908 in provider astro 2024-03-13 08:49:24 proc_lsst.multi:201 [INFO] cancelling 17008908 on provider astro proc_lsst.multi INFO: cancelling 17008908 on provider astro 2024-03-13 08:49:24 proc_lsst.multi:146 [INFO] found job 17008906 in provider astro proc_lsst.multi INFO: found job 17008906 in provider astro 2024-03-13 08:49:24 proc_lsst.multi:201 [INFO] cancelling 17008906 on provider astro proc_lsst.multi INFO: cancelling 17008906 on provider astro 2024-03-13 08:49:25 proc_lsst.multi:146 [INFO] found job 17008903 in provider astro proc_lsst.multi INFO: found job 17008903 in provider astro 2024-03-13 08:49:25 proc_lsst.multi:201 [INFO] cancelling 17008903 on provider astro proc_lsst.multi INFO: cancelling 17008903 on provider astro 2024-03-13 08:49:25 proc_lsst.multi:146 [INFO] found job 17008950 in provider escience_rtx6k proc_lsst.multi INFO: found job 17008950 in provider escience_rtx6k 2024-03-13 08:49:25 proc_lsst.multi:201 [INFO] cancelling 17008950 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17008950 on provider escience_rtx6k 2024-03-13 08:49:25 proc_lsst.multi:146 [INFO] found job 17008951 in provider escience_rtx6k proc_lsst.multi INFO: found job 17008951 in provider escience_rtx6k 2024-03-13 08:49:25 proc_lsst.multi:201 [INFO] cancelling 17008951 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17008951 on provider escience_rtx6k parsl.dataflow.dflow INFO: Shutting down executor multi 2024-03-13 08:49:25 proc_lsst.multi:40 [INFO] Cancelling all provider resources proc_lsst.multi INFO: Cancelling all provider resources 2024-03-13 08:49:25 proc_lsst.multi:47 [INFO] new jobs since last cancel ['17008896', '17008898', '17008899', '17008900', '17008901', '17008902', '17008903', '17008904', '17008905', '17008906', '17008907', '17008908', '17008909', '17008910', '17008911', '17008913', '17008914', '17008915', '17008917', '17008918', '17008919', '17008921', '17008922', '17008923', '17008924', '17008925', '17008926', '17008927', '17008930', '17008934', '17008953', '17008954', '17008961', '17008943', '17008949', '17008950', '17008951', '17008952', '34092'] proc_lsst.multi INFO: new jobs since last cancel ['17008896', '17008898', '17008899', '17008900', '17008901', '17008902', '17008903', '17008904', '17008905', '17008906', '17008907', '17008908', '17008909', '17008910', '17008911', '17008913', '17008914', '17008915', '17008917', '17008918', '17008919', '17008921', '17008922', '17008923', '17008924', '17008925', '17008926', '17008927', '17008930', '17008934', '17008953', '17008954', '17008961', '17008943', '17008949', '17008950', '17008951', '17008952', '34092'] 2024-03-13 08:49:25 proc_lsst.multi:146 [INFO] found job 17008896 in provider astro proc_lsst.multi INFO: found job 17008896 in provider astro 2024-03-13 08:49:25 proc_lsst.multi:201 [INFO] cancelling 17008896 on provider astro proc_lsst.multi INFO: cancelling 17008896 on provider astro 2024-03-13 08:49:25 proc_lsst.multi:146 [INFO] found job 17008898 in provider astro proc_lsst.multi INFO: found job 17008898 in provider astro 2024-03-13 08:49:25 proc_lsst.multi:201 [INFO] cancelling 17008898 on provider astro proc_lsst.multi INFO: cancelling 17008898 on provider astro 2024-03-13 08:49:25 proc_lsst.multi:146 [INFO] found job 17008899 in provider astro proc_lsst.multi INFO: found job 17008899 in provider astro 2024-03-13 08:49:25 proc_lsst.multi:201 [INFO] cancelling 17008899 on provider astro proc_lsst.multi INFO: cancelling 17008899 on provider astro 2024-03-13 08:49:25 proc_lsst.multi:146 [INFO] found job 17008900 in provider astro proc_lsst.multi INFO: found job 17008900 in provider astro 2024-03-13 08:49:25 proc_lsst.multi:201 [INFO] cancelling 17008900 on provider astro proc_lsst.multi INFO: cancelling 17008900 on provider astro 2024-03-13 08:49:25 proc_lsst.multi:146 [INFO] found job 17008901 in provider astro proc_lsst.multi INFO: found job 17008901 in provider astro 2024-03-13 08:49:25 proc_lsst.multi:201 [INFO] cancelling 17008901 on provider astro proc_lsst.multi INFO: cancelling 17008901 on provider astro 2024-03-13 08:49:25 proc_lsst.multi:146 [INFO] found job 17008902 in provider astro proc_lsst.multi INFO: found job 17008902 in provider astro 2024-03-13 08:49:25 proc_lsst.multi:201 [INFO] cancelling 17008902 on provider astro proc_lsst.multi INFO: cancelling 17008902 on provider astro 2024-03-13 08:49:26 proc_lsst.multi:146 [INFO] found job 17008903 in provider astro proc_lsst.multi INFO: found job 17008903 in provider astro 2024-03-13 08:49:26 proc_lsst.multi:201 [INFO] cancelling 17008903 on provider astro proc_lsst.multi INFO: cancelling 17008903 on provider astro 2024-03-13 08:49:26 proc_lsst.multi:146 [INFO] found job 17008904 in provider astro proc_lsst.multi INFO: found job 17008904 in provider astro 2024-03-13 08:49:26 proc_lsst.multi:201 [INFO] cancelling 17008904 on provider astro proc_lsst.multi INFO: cancelling 17008904 on provider astro 2024-03-13 08:49:26 proc_lsst.multi:146 [INFO] found job 17008905 in provider astro proc_lsst.multi INFO: found job 17008905 in provider astro 2024-03-13 08:49:26 proc_lsst.multi:201 [INFO] cancelling 17008905 on provider astro proc_lsst.multi INFO: cancelling 17008905 on provider astro 2024-03-13 08:49:26 proc_lsst.multi:146 [INFO] found job 17008906 in provider astro proc_lsst.multi INFO: found job 17008906 in provider astro 2024-03-13 08:49:26 proc_lsst.multi:201 [INFO] cancelling 17008906 on provider astro proc_lsst.multi INFO: cancelling 17008906 on provider astro 2024-03-13 08:49:26 proc_lsst.multi:146 [INFO] found job 17008907 in provider astro proc_lsst.multi INFO: found job 17008907 in provider astro 2024-03-13 08:49:26 proc_lsst.multi:201 [INFO] cancelling 17008907 on provider astro proc_lsst.multi INFO: cancelling 17008907 on provider astro 2024-03-13 08:49:26 proc_lsst.multi:146 [INFO] found job 17008908 in provider astro proc_lsst.multi INFO: found job 17008908 in provider astro 2024-03-13 08:49:26 proc_lsst.multi:201 [INFO] cancelling 17008908 on provider astro proc_lsst.multi INFO: cancelling 17008908 on provider astro 2024-03-13 08:49:26 proc_lsst.multi:146 [INFO] found job 17008909 in provider astro proc_lsst.multi INFO: found job 17008909 in provider astro 2024-03-13 08:49:26 proc_lsst.multi:201 [INFO] cancelling 17008909 on provider astro proc_lsst.multi INFO: cancelling 17008909 on provider astro 2024-03-13 08:49:26 proc_lsst.multi:146 [INFO] found job 17008910 in provider astro proc_lsst.multi INFO: found job 17008910 in provider astro 2024-03-13 08:49:26 proc_lsst.multi:201 [INFO] cancelling 17008910 on provider astro proc_lsst.multi INFO: cancelling 17008910 on provider astro 2024-03-13 08:49:26 proc_lsst.multi:146 [INFO] found job 17008911 in provider astro proc_lsst.multi INFO: found job 17008911 in provider astro 2024-03-13 08:49:26 proc_lsst.multi:201 [INFO] cancelling 17008911 on provider astro proc_lsst.multi INFO: cancelling 17008911 on provider astro 2024-03-13 08:49:26 proc_lsst.multi:146 [INFO] found job 17008913 in provider astro proc_lsst.multi INFO: found job 17008913 in provider astro 2024-03-13 08:49:26 proc_lsst.multi:201 [INFO] cancelling 17008913 on provider astro proc_lsst.multi INFO: cancelling 17008913 on provider astro 2024-03-13 08:49:27 proc_lsst.multi:146 [INFO] found job 17008914 in provider astro proc_lsst.multi INFO: found job 17008914 in provider astro 2024-03-13 08:49:27 proc_lsst.multi:201 [INFO] cancelling 17008914 on provider astro proc_lsst.multi INFO: cancelling 17008914 on provider astro 2024-03-13 08:49:27 proc_lsst.multi:146 [INFO] found job 17008915 in provider astro proc_lsst.multi INFO: found job 17008915 in provider astro 2024-03-13 08:49:27 proc_lsst.multi:201 [INFO] cancelling 17008915 on provider astro proc_lsst.multi INFO: cancelling 17008915 on provider astro 2024-03-13 08:49:27 proc_lsst.multi:146 [INFO] found job 17008917 in provider astro proc_lsst.multi INFO: found job 17008917 in provider astro 2024-03-13 08:49:27 proc_lsst.multi:201 [INFO] cancelling 17008917 on provider astro proc_lsst.multi INFO: cancelling 17008917 on provider astro 2024-03-13 08:49:27 proc_lsst.multi:146 [INFO] found job 17008918 in provider astro proc_lsst.multi INFO: found job 17008918 in provider astro 2024-03-13 08:49:27 proc_lsst.multi:201 [INFO] cancelling 17008918 on provider astro proc_lsst.multi INFO: cancelling 17008918 on provider astro 2024-03-13 08:49:27 proc_lsst.multi:146 [INFO] found job 17008919 in provider astro proc_lsst.multi INFO: found job 17008919 in provider astro 2024-03-13 08:49:27 proc_lsst.multi:201 [INFO] cancelling 17008919 on provider astro proc_lsst.multi INFO: cancelling 17008919 on provider astro 2024-03-13 08:49:27 proc_lsst.multi:146 [INFO] found job 17008921 in provider astro proc_lsst.multi INFO: found job 17008921 in provider astro 2024-03-13 08:49:27 proc_lsst.multi:201 [INFO] cancelling 17008921 on provider astro proc_lsst.multi INFO: cancelling 17008921 on provider astro 2024-03-13 08:49:27 proc_lsst.multi:146 [INFO] found job 17008922 in provider astro proc_lsst.multi INFO: found job 17008922 in provider astro 2024-03-13 08:49:27 proc_lsst.multi:201 [INFO] cancelling 17008922 on provider astro proc_lsst.multi INFO: cancelling 17008922 on provider astro 2024-03-13 08:49:27 proc_lsst.multi:146 [INFO] found job 17008923 in provider astro proc_lsst.multi INFO: found job 17008923 in provider astro 2024-03-13 08:49:27 proc_lsst.multi:201 [INFO] cancelling 17008923 on provider astro proc_lsst.multi INFO: cancelling 17008923 on provider astro 2024-03-13 08:49:27 proc_lsst.multi:146 [INFO] found job 17008924 in provider astro proc_lsst.multi INFO: found job 17008924 in provider astro 2024-03-13 08:49:27 proc_lsst.multi:201 [INFO] cancelling 17008924 on provider astro proc_lsst.multi INFO: cancelling 17008924 on provider astro 2024-03-13 08:49:27 proc_lsst.multi:146 [INFO] found job 17008925 in provider astro proc_lsst.multi INFO: found job 17008925 in provider astro 2024-03-13 08:49:27 proc_lsst.multi:201 [INFO] cancelling 17008925 on provider astro proc_lsst.multi INFO: cancelling 17008925 on provider astro 2024-03-13 08:49:27 proc_lsst.multi:146 [INFO] found job 17008926 in provider astro proc_lsst.multi INFO: found job 17008926 in provider astro 2024-03-13 08:49:27 proc_lsst.multi:201 [INFO] cancelling 17008926 on provider astro proc_lsst.multi INFO: cancelling 17008926 on provider astro 2024-03-13 08:49:28 proc_lsst.multi:146 [INFO] found job 17008927 in provider astro proc_lsst.multi INFO: found job 17008927 in provider astro 2024-03-13 08:49:28 proc_lsst.multi:201 [INFO] cancelling 17008927 on provider astro proc_lsst.multi INFO: cancelling 17008927 on provider astro 2024-03-13 08:49:28 proc_lsst.multi:146 [INFO] found job 17008930 in provider astro proc_lsst.multi INFO: found job 17008930 in provider astro 2024-03-13 08:49:28 proc_lsst.multi:201 [INFO] cancelling 17008930 on provider astro proc_lsst.multi INFO: cancelling 17008930 on provider astro 2024-03-13 08:49:28 proc_lsst.multi:146 [INFO] found job 17008934 in provider astro proc_lsst.multi INFO: found job 17008934 in provider astro 2024-03-13 08:49:28 proc_lsst.multi:201 [INFO] cancelling 17008934 on provider astro proc_lsst.multi INFO: cancelling 17008934 on provider astro 2024-03-13 08:49:28 proc_lsst.multi:146 [INFO] found job 17008953 in provider escience_a40 proc_lsst.multi INFO: found job 17008953 in provider escience_a40 2024-03-13 08:49:28 proc_lsst.multi:201 [INFO] cancelling 17008953 on provider escience_a40 proc_lsst.multi INFO: cancelling 17008953 on provider escience_a40 2024-03-13 08:49:28 proc_lsst.multi:146 [INFO] found job 17008954 in provider escience_a40 proc_lsst.multi INFO: found job 17008954 in provider escience_a40 2024-03-13 08:49:28 proc_lsst.multi:201 [INFO] cancelling 17008954 on provider escience_a40 proc_lsst.multi INFO: cancelling 17008954 on provider escience_a40 2024-03-13 08:49:28 proc_lsst.multi:146 [INFO] found job 17008961 in provider escience_a40 proc_lsst.multi INFO: found job 17008961 in provider escience_a40 2024-03-13 08:49:28 proc_lsst.multi:201 [INFO] cancelling 17008961 on provider escience_a40 proc_lsst.multi INFO: cancelling 17008961 on provider escience_a40 2024-03-13 08:49:28 proc_lsst.multi:146 [INFO] found job 17008943 in provider escience_rtx6k proc_lsst.multi INFO: found job 17008943 in provider escience_rtx6k 2024-03-13 08:49:28 proc_lsst.multi:201 [INFO] cancelling 17008943 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17008943 on provider escience_rtx6k 2024-03-13 08:49:28 proc_lsst.multi:146 [INFO] found job 17008949 in provider escience_rtx6k proc_lsst.multi INFO: found job 17008949 in provider escience_rtx6k 2024-03-13 08:49:28 proc_lsst.multi:201 [INFO] cancelling 17008949 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17008949 on provider escience_rtx6k 2024-03-13 08:49:28 proc_lsst.multi:146 [INFO] found job 17008950 in provider escience_rtx6k proc_lsst.multi INFO: found job 17008950 in provider escience_rtx6k 2024-03-13 08:49:28 proc_lsst.multi:201 [INFO] cancelling 17008950 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17008950 on provider escience_rtx6k 2024-03-13 08:49:28 proc_lsst.multi:146 [INFO] found job 17008951 in provider escience_rtx6k proc_lsst.multi INFO: found job 17008951 in provider escience_rtx6k 2024-03-13 08:49:28 proc_lsst.multi:201 [INFO] cancelling 17008951 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17008951 on provider escience_rtx6k 2024-03-13 08:49:28 proc_lsst.multi:146 [INFO] found job 17008952 in provider escience_rtx6k proc_lsst.multi INFO: found job 17008952 in provider escience_rtx6k 2024-03-13 08:49:28 proc_lsst.multi:201 [INFO] cancelling 17008952 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17008952 on provider escience_rtx6k 2024-03-13 08:49:29 proc_lsst.multi:146 [INFO] found job 34092 in provider local proc_lsst.multi INFO: found job 34092 in provider local 2024-03-13 08:49:29 proc_lsst.multi:201 [INFO] cancelling 34092 on provider local proc_lsst.multi INFO: cancelling 34092 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 34092 parsl.providers.local.local WARNING: Failed to kill PID: 34092 and child processes on local 2024-03-13 08:49:30 proc_lsst.multi:50 [INFO] no new jobs since last cancel, resuming executor shutdown proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt parsl.dataflow.dflow INFO: Shut down executor multi parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete parsl.executors.threads DEBUG: Done with executor shutdown parsl.dataflow.dflow INFO: Shut down executor _parsl_internal parsl.dataflow.dflow INFO: Terminated executors parsl.dataflow.dflow INFO: DFK cleanup complete parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 293.0355 seconds lsst.ctrl.bps.drivers INFO: Run 'DEEP_20190926_B1a_science#step1_20240313T084347Z' submitted for execution with id 'None' lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 293.0415 seconds; current memory usage: 0.352 Gibyte, delta: 0.006 Gibyte, peak delta: 0.006 Gibyte lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 306.9559 seconds; current memory usage: 0.352 Gibyte, delta: 0.213 Gibyte, peak delta: 0.213 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.352 Gibyte (main), 0.352 Gibyte (largest child process) Run Id: None Run Name: DEEP_20190926_B1a_science#step1_20240313T084347Z parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up real 5m9.143s user 0m47.014s sys 0m20.361s