+ PROC_LSST_SUBMIT_PATH=/mmfs1/home/stevengs/dirac/DEEP/submit + bps submit /gscratch/dirac/shared/opt/proc_lsst/pipelines/submit.yaml -b /mmfs1/home/stevengs/dirac/DEEP/repo -i DEEP/20190403/A0c --output-run DEEP/20190403/A0c/science#step1/20240313T001343Z --qgraph pipeline.qgraph lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process. lsst.ctrl.bps.drivers INFO: Starting submission process lsst.ctrl.bps.drivers INFO: Initializing execution environment lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 2.6745 seconds; current memory usage: 0.178 Gibyte, delta: 0.041 Gibyte, peak delta: 0.054 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.190 Gibyte (main), 0.000 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph) lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.0048 seconds lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.0097 seconds lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/pipeline_orig.qgraph DEEP/20190403/A0c/science#step1/20240313T001343Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/pipeline.qgraph lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 5.8539 seconds lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 14.1875 seconds; current memory usage: 0.341 Gibyte, delta: 0.163 Gibyte, peak delta: 0.150 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.341 Gibyte (main), 0.332 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs) lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0161 seconds; current memory usage: 0.341 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.341 Gibyte (main), 0.332 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 208 cluster(s) lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow) lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20190403_A0c_science#step1_20240313T001343Z' lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 0.0799 seconds; current memory usage: 0.341 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.341 Gibyte (main), 0.332 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 210 job(s) (including final) lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow) parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2 Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses s_addresses.add(address_by_interface(interface)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper retval = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface return socket.inet_ntoa(fcntl.ioctl( ^^^^^^^^^^^^ OSError: [Errno 99] Cannot assign requested address lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 0.2245 seconds; current memory usage: 0.342 Gibyte, delta: 0.001 Gibyte, peak delta: 0.001 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.342 Gibyte (main), 0.332 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting submit stage lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000 parsl.dataflow.dflow INFO: Starting DataFlowKernel with config Config( app_cache=True, checkpoint_files=None, checkpoint_mode='task_exit', checkpoint_period=None, executors=[MultiHighThroughputExecutor()], garbage_collect=True, initialize_logging=True, internal_tasks_max_threads=10, max_idletime=120.0, monitoring=None, retries=1, retry_handler=None, run_dir='runinfo', strategy='simple', usage_tracking=False ) parsl.dataflow.dflow INFO: Parsl version: 2023.06.12 parsl.usage_tracking.usage DEBUG: Tracking status: False parsl.dataflow.dflow INFO: Run id is: 44f0de27-1152-49d6-a92f-1d47bfdf7f52 parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name parsl.dataflow.memoization INFO: App caching initialized parsl.dataflow.strategy DEBUG: Scaling strategy: simple parsl.executors.high_throughput.executor DEBUG: Starting queue management thread parsl.executors.high_throughput.executor DEBUG: queue management worker starting parsl.executors.high_throughput.executor DEBUG: Started queue management thread Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z parsl.executors.high_throughput.executor DEBUG: Created management thread: parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id={block_id} --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider: parsl.executors.status_handling INFO: Scaling out by 1 blocks parsl.executors.status_handling INFO: Allocated block ID 0 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0 2024-03-13 00:15:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 2024-03-13 00:15:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 00:15:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-13 00:15:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 00:15:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local parsl.providers.local.local DEBUG: Launching in remote mode 2024-03-13 00:15:10 proc_lsst.multi:170 [INFO] [multi] job_id 10228 proc_lsst.multi INFO: [multi] job_id 10228 2024-03-13 00:15:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 00:15:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 00:15:10 proc_lsst.multi:178 [INFO] [multi] provider local accepted submit and returned 10228 proc_lsst.multi INFO: [multi] provider local accepted submit and returned 10228 parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 10228 parsl.dataflow.job_status_poller DEBUG: Adding executor multi parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 0 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 0 has memoization hash dfcbb6858bd62a02b1c647f383fac018 parsl.dataflow.memoization INFO: Task 0 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9938540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1 parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845889/47baa340-e987-4003-a9f7-a3eed82a19d4_characterizeImage_845889_2.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845889/47baa340-e987-4003-a9f7-a3eed82a19d4_characterizeImage_845889_2.stderr parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1 submitted for App calibrate, waiting on task 0 parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2 has memoization hash 74965c9fb0108800374f15fc3f8aad54 parsl.dataflow.memoization INFO: Task 2 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9938cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 2 parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845969/e5463690-4c90-49b8-bf95-413e46148a08_characterizeImage_845969_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845969/e5463690-4c90-49b8-bf95-413e46148a08_characterizeImage_845969_2.stderr parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3 submitted for App calibrate, waiting on task 2 parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 3 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash 9d87fd5e64d591d853b837f65776411a parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9938fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 3 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845923/307bb778-9554-4cf5-87a7-59a661125856_characterizeImage_845923_2.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845923/307bb778-9554-4cf5-87a7-59a661125856_characterizeImage_845923_2.stderr parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5 submitted for App calibrate, waiting on task 4 parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 5 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 6 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 6 has memoization hash 88e67c74e0c27846fbc6159ad3abc379 parsl.dataflow.memoization INFO: Task 6 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9939080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 4 parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845917/6af4a453-83c9-42cf-b400-c9684629b977_characterizeImage_845917_2.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845917/6af4a453-83c9-42cf-b400-c9684629b977_characterizeImage_845917_2.stderr parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 7 submitted for App calibrate, waiting on task 6 parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 7 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 8 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 8 has memoization hash 0a8648caf6226ed5ca02cccbe0a7329a parsl.dataflow.memoization INFO: Task 8 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9939120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 5 parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845879/9444aec8-7f5a-4bb7-abb6-e19832d51831_characterizeImage_845879_2.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845879/9444aec8-7f5a-4bb7-abb6-e19832d51831_characterizeImage_845879_2.stderr parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 9 submitted for App calibrate, waiting on task 8 parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 9 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 10 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash 0e239dd7fa281aaaa938ba9801fea216 parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9939300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 6 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845956/9dd128fe-62fa-4ea8-859e-c6327748c2b1_characterizeImage_845956_2.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845956/9dd128fe-62fa-4ea8-859e-c6327748c2b1_characterizeImage_845956_2.stderr parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 11 submitted for App calibrate, waiting on task 10 parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 11 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 12 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 12 has memoization hash 98ac88c7618fcf1413ef0530b9ef8734 parsl.dataflow.memoization INFO: Task 12 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99396c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 7 parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845925/932d1e2c-a64f-4d9f-bf21-e4e5d4cff2fc_characterizeImage_845925_2.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845925/932d1e2c-a64f-4d9f-bf21-e4e5d4cff2fc_characterizeImage_845925_2.stderr parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 13 submitted for App calibrate, waiting on task 12 parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 13 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 14 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 14 has memoization hash 3159d685790ff3c487a08c113df38067 parsl.dataflow.memoization INFO: Task 14 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9939940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 8 parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845876/b79d0412-b23f-4fc3-9028-4c3becfc27d5_characterizeImage_845876_2.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845876/b79d0412-b23f-4fc3-9028-4c3becfc27d5_characterizeImage_845876_2.stderr parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 15 submitted for App calibrate, waiting on task 14 parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 15 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 16 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash cf4b320082b76b1e2fe672aab84cc0ec parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9938860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 9 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845955/680a196a-21f7-494d-bc9c-ca5d4d30c999_characterizeImage_845955_2.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845955/680a196a-21f7-494d-bc9c-ca5d4d30c999_characterizeImage_845955_2.stderr parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 17 submitted for App calibrate, waiting on task 16 parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 17 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 18 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 18 has memoization hash e996bdc0ebe1e7c59b4fc3cd6b3bd534 parsl.dataflow.memoization INFO: Task 18 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9939c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 10 parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845951/4cab1fe7-79b5-4f3d-8f17-0b8b0e1dea22_characterizeImage_845951_2.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845951/4cab1fe7-79b5-4f3d-8f17-0b8b0e1dea22_characterizeImage_845951_2.stderr parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 19 submitted for App calibrate, waiting on task 18 parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 19 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 20 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 20 has memoization hash 77ae5deac8c2a3a9f7fac6e9a5f42fbf parsl.dataflow.memoization INFO: Task 20 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9939e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 11 parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845949/f3355e5d-55be-4f4a-883f-ade688be4595_characterizeImage_845949_2.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845949/f3355e5d-55be-4f4a-883f-ade688be4595_characterizeImage_845949_2.stderr parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 21 submitted for App calibrate, waiting on task 20 parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 21 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 22 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash fbcfea2073e27e6eda147bae35d01516 parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9939da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 12 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845962/9a3ed4f5-fec4-4bf8-80ff-9071e87fee48_characterizeImage_845962_2.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845962/9a3ed4f5-fec4-4bf8-80ff-9071e87fee48_characterizeImage_845962_2.stderr parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 23 submitted for App calibrate, waiting on task 22 parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 23 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 24 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 24 has memoization hash 18e2ca751e16fca96e811f0c522e28ba parsl.dataflow.memoization INFO: Task 24 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de993a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 13 parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845929/1a960c36-216f-4b57-b1f9-5a6150aadd7a_characterizeImage_845929_2.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845929/1a960c36-216f-4b57-b1f9-5a6150aadd7a_characterizeImage_845929_2.stderr parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 25 submitted for App calibrate, waiting on task 24 parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 25 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 26 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 26 has memoization hash 048d028a81ac065b28de0c859d43adb6 parsl.dataflow.memoization INFO: Task 26 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de993b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 14 parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845924/ec22148e-7dcf-487c-85ce-66bade631426_characterizeImage_845924_2.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845924/ec22148e-7dcf-487c-85ce-66bade631426_characterizeImage_845924_2.stderr parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 27 submitted for App calibrate, waiting on task 26 parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 27 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 28 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash 719d7be68431e19edca9463945dbee24 parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de993b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 15 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845895/c2b6501b-7148-4734-8d01-d113a76b8478_characterizeImage_845895_2.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845895/c2b6501b-7148-4734-8d01-d113a76b8478_characterizeImage_845895_2.stderr parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 29 submitted for App calibrate, waiting on task 28 parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 29 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 30 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 30 has memoization hash ba2ae0daa7158fc9112c89a875f03037 parsl.dataflow.memoization INFO: Task 30 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de993b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 16 parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845906/6892a379-1d64-46ab-9ec4-ce60e4af68dc_characterizeImage_845906_2.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845906/6892a379-1d64-46ab-9ec4-ce60e4af68dc_characterizeImage_845906_2.stderr parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 31 submitted for App calibrate, waiting on task 30 parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 31 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 32 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 32 has memoization hash 469fb8ae935bab5e16cecb86e5f5d753 parsl.dataflow.memoization INFO: Task 32 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de993b2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 17 parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845909/ac020887-2a2a-4cd3-973d-f2cf3d0ca195_characterizeImage_845909_2.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845909/ac020887-2a2a-4cd3-973d-f2cf3d0ca195_characterizeImage_845909_2.stderr parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 33 submitted for App calibrate, waiting on task 32 parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 33 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 34 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash c01c083e32fcf85f4e300591a01b3000 parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de993af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 18 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845938/04e8f597-23e1-4218-883d-b3fc7d9b8399_characterizeImage_845938_2.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845938/04e8f597-23e1-4218-883d-b3fc7d9b8399_characterizeImage_845938_2.stderr parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 35 submitted for App calibrate, waiting on task 34 parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 35 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 36 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 36 has memoization hash e5f44513b16c88757ba89a5eba595551 parsl.dataflow.memoization INFO: Task 36 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de993b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 19 parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845888/6662b1fa-0e7e-445c-94b7-c3e176b4f763_characterizeImage_845888_2.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845888/6662b1fa-0e7e-445c-94b7-c3e176b4f763_characterizeImage_845888_2.stderr parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 37 submitted for App calibrate, waiting on task 36 parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 37 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 38 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 38 has memoization hash f64e69ad7b5d5c9b4323b3d25b5f347f parsl.dataflow.memoization INFO: Task 38 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de993ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 20 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845920/38b5da06-64ee-4e25-965b-c964097db327_characterizeImage_845920_2.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845920/38b5da06-64ee-4e25-965b-c964097db327_characterizeImage_845920_2.stderr parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 39 submitted for App calibrate, waiting on task 38 parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 39 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 40 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash fed050588955d9cdeb4e4959aef78ddf parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de993ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 21 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845926/5a3176b1-cf38-4e9b-9585-03024c203f7e_characterizeImage_845926_2.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845926/5a3176b1-cf38-4e9b-9585-03024c203f7e_characterizeImage_845926_2.stderr parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 41 submitted for App calibrate, waiting on task 40 parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 41 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 42 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 42 has memoization hash 7ba2785931ac668f5aa3083a3524de72 parsl.dataflow.memoization INFO: Task 42 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de993a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 22 parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845916/797024ed-829b-4ed2-9eaf-db520235adbf_characterizeImage_845916_2.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845916/797024ed-829b-4ed2-9eaf-db520235adbf_characterizeImage_845916_2.stderr parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 43 submitted for App calibrate, waiting on task 42 parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 43 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 44 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 44 has memoization hash 070ff849bee17ce25704e41606eee33c parsl.dataflow.memoization INFO: Task 44 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de993a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 23 parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845954/eaf6cc58-5cb9-4477-924a-e33e022dcc7f_characterizeImage_845954_2.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845954/eaf6cc58-5cb9-4477-924a-e33e022dcc7f_characterizeImage_845954_2.stderr parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 45 submitted for App calibrate, waiting on task 44 parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 45 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 46 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 46 has memoization hash e890f1b1d6866bc6c740aa0eec7e066e parsl.dataflow.memoization INFO: Task 46 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de993a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 24 parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845886/9251d4db-ca14-412f-92e9-66fdd14198d5_characterizeImage_845886_2.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845886/9251d4db-ca14-412f-92e9-66fdd14198d5_characterizeImage_845886_2.stderr parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 47 submitted for App calibrate, waiting on task 46 parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 47 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 48 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 48 has memoization hash 017a71140c924a8d1120a2a44be86333 parsl.dataflow.memoization INFO: Task 48 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de993b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 25 parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845891/cd95ec31-d06c-4bdb-a6c3-642fc6d57370_characterizeImage_845891_2.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845891/cd95ec31-d06c-4bdb-a6c3-642fc6d57370_characterizeImage_845891_2.stderr parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 49 submitted for App calibrate, waiting on task 48 parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 49 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 50 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 50 has memoization hash d04a3d9d8240c79d5986e53c79af23f9 parsl.dataflow.memoization INFO: Task 50 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de993bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 26 parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845953/2bd41c30-2e53-4ce5-9fa8-8eae6750494d_characterizeImage_845953_2.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845953/2bd41c30-2e53-4ce5-9fa8-8eae6750494d_characterizeImage_845953_2.stderr parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 51 submitted for App calibrate, waiting on task 50 parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 51 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 52 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash 58e47534fd0397325e9e00c6d86effed parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de993bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 27 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845893/52c3beab-ecdd-4b0e-9db2-f6b12d2eb801_characterizeImage_845893_2.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845893/52c3beab-ecdd-4b0e-9db2-f6b12d2eb801_characterizeImage_845893_2.stderr parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 53 submitted for App calibrate, waiting on task 52 parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 53 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 54 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 54 has memoization hash b5008116ffc61a6f3007ebe16fda163e parsl.dataflow.memoization INFO: Task 54 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99680e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 28 parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845927/86c93158-c0d4-4a62-8934-37cd77914d58_characterizeImage_845927_2.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845927/86c93158-c0d4-4a62-8934-37cd77914d58_characterizeImage_845927_2.stderr parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 55 submitted for App calibrate, waiting on task 54 parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 55 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 56 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 56 has memoization hash f1f634f78f5dbf3f26226408456d4069 parsl.dataflow.memoization INFO: Task 56 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9968180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 29 parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845928/ea42931b-5e2c-48e0-908f-ad9a56071956_characterizeImage_845928_2.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845928/ea42931b-5e2c-48e0-908f-ad9a56071956_characterizeImage_845928_2.stderr parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 57 submitted for App calibrate, waiting on task 56 parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 57 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 58 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 58 has memoization hash 16680ccd761e9f2b2599a7273b360be1 parsl.dataflow.memoization INFO: Task 58 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99682c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 30 parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845892/f31b5b11-e4a4-4d7d-a8a9-d6d25d84966c_characterizeImage_845892_2.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845892/f31b5b11-e4a4-4d7d-a8a9-d6d25d84966c_characterizeImage_845892_2.stderr parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 59 submitted for App calibrate, waiting on task 58 parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 59 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 60 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 60 has memoization hash 251cbc3ea7dee02d57b87bb8c701f3c7 parsl.dataflow.memoization INFO: Task 60 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99684a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 31 parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845913/024ff1de-e31d-4139-b3c5-b73c055d82ca_characterizeImage_845913_2.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845913/024ff1de-e31d-4139-b3c5-b73c055d82ca_characterizeImage_845913_2.stderr parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 61 submitted for App calibrate, waiting on task 60 parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 61 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 62 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 62 has memoization hash d8a281c1b3d9f765dac77891d60e22e5 parsl.dataflow.memoization INFO: Task 62 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9968680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 32 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845939/b78c437e-0357-4a72-b7e7-e92ce8ff32e7_characterizeImage_845939_2.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845939/b78c437e-0357-4a72-b7e7-e92ce8ff32e7_characterizeImage_845939_2.stderr parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 63 submitted for App calibrate, waiting on task 62 parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 63 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 64 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash 292107de6482feee20e278a818f6538c parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9968a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 33 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845874/4badbe55-5bf8-4c60-9092-cb226d85eda1_characterizeImage_845874_2.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845874/4badbe55-5bf8-4c60-9092-cb226d85eda1_characterizeImage_845874_2.stderr parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 65 submitted for App calibrate, waiting on task 64 parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 65 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 66 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 66 has memoization hash 95b9654e07da090311c0189aef5b5d18 parsl.dataflow.memoization INFO: Task 66 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9968860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 34 parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845975/da1c7d8f-e6f9-46a6-b3f2-8258eaeb2b51_characterizeImage_845975_2.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845975/da1c7d8f-e6f9-46a6-b3f2-8258eaeb2b51_characterizeImage_845975_2.stderr parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 67 submitted for App calibrate, waiting on task 66 parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 67 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 68 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 68 has memoization hash d8ec5db9d7aa4d44b438da3d92317e06 parsl.dataflow.memoization INFO: Task 68 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9968c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 35 parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845945/3a266ace-a6aa-428e-88a7-b3fccd7d2042_characterizeImage_845945_2.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845945/3a266ace-a6aa-428e-88a7-b3fccd7d2042_characterizeImage_845945_2.stderr parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 69 submitted for App calibrate, waiting on task 68 parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 69 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 70 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash 59111f98da0a7bd3e808682dc15c3e52 parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9968e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 36 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845967/5a555250-817e-4342-92e0-7aaa9746b09c_characterizeImage_845967_2.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845967/5a555250-817e-4342-92e0-7aaa9746b09c_characterizeImage_845967_2.stderr parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 71 submitted for App calibrate, waiting on task 70 parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 71 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 72 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 72 has memoization hash 78202a5dd2e162448f52d6908381c4c3 parsl.dataflow.memoization INFO: Task 72 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9968fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 37 parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845918/0b009a5e-e046-4261-8fa5-3c38569211ba_characterizeImage_845918_2.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845918/0b009a5e-e046-4261-8fa5-3c38569211ba_characterizeImage_845918_2.stderr parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 73 submitted for App calibrate, waiting on task 72 parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 73 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 74 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 74 has memoization hash 54617f18a55d67a4158e9f7f5c124cce parsl.dataflow.memoization INFO: Task 74 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99693a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 38 parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845964/303de284-995e-41c2-8c35-f444be467aaa_characterizeImage_845964_2.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845964/303de284-995e-41c2-8c35-f444be467aaa_characterizeImage_845964_2.stderr parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 75 submitted for App calibrate, waiting on task 74 parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 75 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 76 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash 6731d816e8d8239d3dbf0561c8423f22 parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99691c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 39 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845885/af1fe9c9-54fc-473f-887d-3a7ae72aa8a3_characterizeImage_845885_2.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845885/af1fe9c9-54fc-473f-887d-3a7ae72aa8a3_characterizeImage_845885_2.stderr parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 77 submitted for App calibrate, waiting on task 76 parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 77 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 78 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 78 has memoization hash fe49a52810bd6bb64f918aabbddc662a parsl.dataflow.memoization INFO: Task 78 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9969580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 40 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845941/0c7af297-2442-4256-b0b6-1c9c478a6c29_characterizeImage_845941_2.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845941/0c7af297-2442-4256-b0b6-1c9c478a6c29_characterizeImage_845941_2.stderr parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 79 submitted for App calibrate, waiting on task 78 parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 79 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 80 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 80 has memoization hash b09277256f23afd0bad289210c29570a parsl.dataflow.memoization INFO: Task 80 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9969760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 41 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845957/7aa5b53f-f329-414b-9b47-d7b6f9e13666_characterizeImage_845957_2.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845957/7aa5b53f-f329-414b-9b47-d7b6f9e13666_characterizeImage_845957_2.stderr parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 81 submitted for App calibrate, waiting on task 80 parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 81 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 82 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash 571deec6cb8c2b574c780b01457b7ed0 parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9969940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 42 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845883/ea7a2c77-8df1-498a-aa9e-940279db3652_characterizeImage_845883_2.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845883/ea7a2c77-8df1-498a-aa9e-940279db3652_characterizeImage_845883_2.stderr parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 83 submitted for App calibrate, waiting on task 82 parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 83 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 84 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 84 has memoization hash 5edcb9752b5fda47225a99f5e0a8d9c8 parsl.dataflow.memoization INFO: Task 84 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9969d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 43 parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845919/f5d0cb70-b110-4afa-b2a2-4a6f4f1450cf_characterizeImage_845919_2.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845919/f5d0cb70-b110-4afa-b2a2-4a6f4f1450cf_characterizeImage_845919_2.stderr parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 85 submitted for App calibrate, waiting on task 84 parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 85 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 86 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 86 has memoization hash bea4af1b9022478ce8f2cd1c66f0db0c parsl.dataflow.memoization INFO: Task 86 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9969b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 44 parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845936/0ad24b81-78f0-4a71-85b1-5897243e3959_characterizeImage_845936_2.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845936/0ad24b81-78f0-4a71-85b1-5897243e3959_characterizeImage_845936_2.stderr parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 87 submitted for App calibrate, waiting on task 86 parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 87 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 88 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 88 has memoization hash 41840f84335b33e2575117574502b150 parsl.dataflow.memoization INFO: Task 88 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9969ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 45 parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845947/f101aedf-9483-455c-b975-a91bbd2761c6_characterizeImage_845947_2.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845947/f101aedf-9483-455c-b975-a91bbd2761c6_characterizeImage_845947_2.stderr parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 89 submitted for App calibrate, waiting on task 88 parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 89 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 90 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 90 has memoization hash 35cad0d2e8de02aa550181216d5111dc parsl.dataflow.memoization INFO: Task 90 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de996a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 46 parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845896/b046a7ec-89a9-45b3-ab8f-876092c79520_characterizeImage_845896_2.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845896/b046a7ec-89a9-45b3-ab8f-876092c79520_characterizeImage_845896_2.stderr parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 91 submitted for App calibrate, waiting on task 90 parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 91 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 92 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash 1a700a3cc267ef0bf4016ba03e69a613 parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de996a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 47 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845907/cffcd057-7872-437d-9715-76380aa988d4_characterizeImage_845907_2.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845907/cffcd057-7872-437d-9715-76380aa988d4_characterizeImage_845907_2.stderr parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 93 submitted for App calibrate, waiting on task 92 parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 93 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 94 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 94 has memoization hash 1152693677a0369f82a87ee6d585cc04 parsl.dataflow.memoization INFO: Task 94 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de996a660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 48 parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845899/56e7a245-664c-40c0-bd2b-24779e8ad2e0_characterizeImage_845899_2.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845899/56e7a245-664c-40c0-bd2b-24779e8ad2e0_characterizeImage_845899_2.stderr parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 95 submitted for App calibrate, waiting on task 94 parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 95 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 96 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 96 has memoization hash d970371a5a4b416dc608b556d1ed8ed2 parsl.dataflow.memoization INFO: Task 96 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de996a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 49 parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845908/2cccf77c-c5bd-400f-b9f6-e902f3419685_characterizeImage_845908_2.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845908/2cccf77c-c5bd-400f-b9f6-e902f3419685_characterizeImage_845908_2.stderr parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 97 submitted for App calibrate, waiting on task 96 parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 97 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 98 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 98 has memoization hash bba3b5a3efc393e35738df156d37271f parsl.dataflow.memoization INFO: Task 98 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de996a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 50 parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845958/d60f3d61-9a95-457d-97fe-f040390eb6f8_characterizeImage_845958_2.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845958/d60f3d61-9a95-457d-97fe-f040390eb6f8_characterizeImage_845958_2.stderr parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 99 submitted for App calibrate, waiting on task 98 parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 99 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 100 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash 2a005eae06a1644529e6f392ef4ef4c1 parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de996aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 51 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845880/abd6192e-1c66-4408-9e4c-dec63d0b40ab_characterizeImage_845880_2.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845880/abd6192e-1c66-4408-9e4c-dec63d0b40ab_characterizeImage_845880_2.stderr parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 101 submitted for App calibrate, waiting on task 100 parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 101 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 102 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 102 has memoization hash b17a8e6300c09938af380732f8a8f6e4 parsl.dataflow.memoization INFO: Task 102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de996ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 52 parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845961/7727537b-7d07-45a6-9122-acbcb9e077ce_characterizeImage_845961_2.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845961/7727537b-7d07-45a6-9122-acbcb9e077ce_characterizeImage_845961_2.stderr parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 103 submitted for App calibrate, waiting on task 102 parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 103 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 104 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 104 has memoization hash 26a197c55a718db18c438ed6ab2c43d0 parsl.dataflow.memoization INFO: Task 104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de996afc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 53 parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845946/1a01a87b-7ea3-4ff9-97de-18786cddcb4e_characterizeImage_845946_2.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845946/1a01a87b-7ea3-4ff9-97de-18786cddcb4e_characterizeImage_845946_2.stderr parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 105 submitted for App calibrate, waiting on task 104 parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 105 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 106 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash 56f9b34c1d6d2fae87614ce08fb1d76a parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de996ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 54 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845897/db4180e0-e7ad-4ce2-a1f2-4ce12f40d7d5_characterizeImage_845897_2.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845897/db4180e0-e7ad-4ce2-a1f2-4ce12f40d7d5_characterizeImage_845897_2.stderr parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 107 submitted for App calibrate, waiting on task 106 parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 107 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 108 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 108 has memoization hash b21401054395f5f8ba48adc9410115f2 parsl.dataflow.memoization INFO: Task 108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de996b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 55 parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845890/2b57e739-0b5a-4f52-a6dd-18fe97769482_characterizeImage_845890_2.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845890/2b57e739-0b5a-4f52-a6dd-18fe97769482_characterizeImage_845890_2.stderr parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 109 submitted for App calibrate, waiting on task 108 parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 109 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 110 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 110 has memoization hash 1b4d13183092be934a16bef0db0574f0 parsl.dataflow.memoization INFO: Task 110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de996b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 56 parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845934/6d57084e-9f7b-4001-a687-2e72889d80fe_characterizeImage_845934_2.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845934/6d57084e-9f7b-4001-a687-2e72889d80fe_characterizeImage_845934_2.stderr parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 111 submitted for App calibrate, waiting on task 110 parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 111 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 112 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 112 has memoization hash 2937485724674bc21461a4ffe8d183ff parsl.dataflow.memoization INFO: Task 112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de996b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 57 parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845901/238275a5-1580-4fe5-874f-03519b7a35f4_characterizeImage_845901_2.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845901/238275a5-1580-4fe5-874f-03519b7a35f4_characterizeImage_845901_2.stderr parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 113 submitted for App calibrate, waiting on task 112 parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 113 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 114 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 114 has memoization hash 05e5a6aaca09e363ebff001ce54648f8 parsl.dataflow.memoization INFO: Task 114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de996b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 58 parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845965/c6c0e87d-e3fc-4858-bef7-c19e9f1aea39_characterizeImage_845965_2.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845965/c6c0e87d-e3fc-4858-bef7-c19e9f1aea39_characterizeImage_845965_2.stderr parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 115 submitted for App calibrate, waiting on task 114 parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 115 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 116 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 116 has memoization hash 58a5f1323cffab8aba0b198436379bbf parsl.dataflow.memoization INFO: Task 116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de996b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 59 parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845960/0c21b0b0-2e4e-4feb-9471-2c58334fc3d3_characterizeImage_845960_2.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845960/0c21b0b0-2e4e-4feb-9471-2c58334fc3d3_characterizeImage_845960_2.stderr parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 117 submitted for App calibrate, waiting on task 116 parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 117 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 118 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash f1502731263e78151067c062569940af parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de996bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 60 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845921/5199ae07-c639-49d5-8195-de5cfcf37276_characterizeImage_845921_2.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845921/5199ae07-c639-49d5-8195-de5cfcf37276_characterizeImage_845921_2.stderr parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 119 submitted for App calibrate, waiting on task 118 parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 119 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 120 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 120 has memoization hash 39fa3e7c682bb37e7b54f33bc6e4fc55 parsl.dataflow.memoization INFO: Task 120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de996bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 61 parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845944/a6044269-9d83-4d2b-b2c5-0d2f8fd837c1_characterizeImage_845944_2.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845944/a6044269-9d83-4d2b-b2c5-0d2f8fd837c1_characterizeImage_845944_2.stderr parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 121 submitted for App calibrate, waiting on task 120 parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 121 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 122 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 122 has memoization hash 8be815d32965502f9ba5eaf4f1f0e28c parsl.dataflow.memoization INFO: Task 122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9994040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 62 parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845931/24c0bb7d-7496-4719-a33c-c8419935b473_characterizeImage_845931_2.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845931/24c0bb7d-7496-4719-a33c-c8419935b473_characterizeImage_845931_2.stderr parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 123 submitted for App calibrate, waiting on task 122 parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 123 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 124 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 124 has memoization hash 28c8cdbabcc323c8e97ebf56b860f027 parsl.dataflow.memoization INFO: Task 124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99942c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 63 parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845943/3d4b9af2-e295-4738-98ac-d7a871c68578_characterizeImage_845943_2.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845943/3d4b9af2-e295-4738-98ac-d7a871c68578_characterizeImage_845943_2.stderr parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 125 submitted for App calibrate, waiting on task 124 parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 125 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 126 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 126 has memoization hash 634b7baf979a5f735559d77d3a9ec9ca parsl.dataflow.memoization INFO: Task 126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99940e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 64 parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845894/310d287a-0551-48bb-8f2d-6afe4b4d8367_characterizeImage_845894_2.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845894/310d287a-0551-48bb-8f2d-6afe4b4d8367_characterizeImage_845894_2.stderr parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 127 submitted for App calibrate, waiting on task 126 parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 127 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 128 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 128 has memoization hash da349ef6b94b0ceaf780a692c10ee37a parsl.dataflow.memoization INFO: Task 128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99944a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 65 parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845902/33330bca-6bd1-4857-85da-515b4f803089_characterizeImage_845902_2.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845902/33330bca-6bd1-4857-85da-515b4f803089_characterizeImage_845902_2.stderr parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 129 submitted for App calibrate, waiting on task 128 parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 129 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 130 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 130 has memoization hash d7977db724d22a2451ef2d979c1a6c27 parsl.dataflow.memoization INFO: Task 130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9994680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 66 parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845915/6203f0d5-9054-4686-8b95-faef0a5edb06_characterizeImage_845915_2.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845915/6203f0d5-9054-4686-8b95-faef0a5edb06_characterizeImage_845915_2.stderr parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 131 submitted for App calibrate, waiting on task 130 parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 131 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 132 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 132 has memoization hash 5f1f9fea53aa5aeb4a8a20beb0ce6740 parsl.dataflow.memoization INFO: Task 132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9994860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 67 parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845910/b6a1b982-bd05-4e4f-99e1-3adbfca0c6aa_characterizeImage_845910_2.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845910/b6a1b982-bd05-4e4f-99e1-3adbfca0c6aa_characterizeImage_845910_2.stderr parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 133 submitted for App calibrate, waiting on task 132 parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 133 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 134 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 134 has memoization hash 4cb6affb9a82af1e1e6752ca4a88faf0 parsl.dataflow.memoization INFO: Task 134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9994c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 68 parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845933/e24cc46c-28c5-4500-9354-c09907f1ab1a_characterizeImage_845933_2.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845933/e24cc46c-28c5-4500-9354-c09907f1ab1a_characterizeImage_845933_2.stderr parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 135 submitted for App calibrate, waiting on task 134 parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 135 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 136 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 136 has memoization hash fa8677c0e62d828621eed9abfac2961e parsl.dataflow.memoization INFO: Task 136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9994a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 69 parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845873/c4125fac-ad41-4766-890a-5eb55c5d1ff6_characterizeImage_845873_2.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845873/c4125fac-ad41-4766-890a-5eb55c5d1ff6_characterizeImage_845873_2.stderr parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 137 submitted for App calibrate, waiting on task 136 parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 137 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 138 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 138 has memoization hash a464cc6d36b1e40e30e451fcabdd05fb parsl.dataflow.memoization INFO: Task 138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9994e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 70 parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845942/654704d2-6c3f-4dd2-a260-0a17b3b3d2ad_characterizeImage_845942_2.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845942/654704d2-6c3f-4dd2-a260-0a17b3b3d2ad_characterizeImage_845942_2.stderr parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 139 submitted for App calibrate, waiting on task 138 parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 139 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 140 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 140 has memoization hash 2efcf3d6aaab8c7d58eab75596239da6 parsl.dataflow.memoization INFO: Task 140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9994fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 71 parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845970/b5026a35-dc05-4010-b148-df5c3b5770b0_characterizeImage_845970_2.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845970/b5026a35-dc05-4010-b148-df5c3b5770b0_characterizeImage_845970_2.stderr parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 141 submitted for App calibrate, waiting on task 140 parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 141 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 142 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 142 has memoization hash 00b56b1de7658aa4bc1562e0d4c859fc parsl.dataflow.memoization INFO: Task 142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99951c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 72 parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845971/85c8671d-b56e-4084-b9a7-e0c7fc16a71d_characterizeImage_845971_2.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845971/85c8671d-b56e-4084-b9a7-e0c7fc16a71d_characterizeImage_845971_2.stderr parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 143 submitted for App calibrate, waiting on task 142 parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 143 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 144 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 144 has memoization hash af259f7fcd1ef5dfbc0433c91cc0a582 parsl.dataflow.memoization INFO: Task 144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9995580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 73 parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845940/8242da42-4187-4773-a9dd-eaa55e453baa_characterizeImage_845940_2.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845940/8242da42-4187-4773-a9dd-eaa55e453baa_characterizeImage_845940_2.stderr parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 145 submitted for App calibrate, waiting on task 144 parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 145 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 146 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 146 has memoization hash 8e4689bdd3a43f6074fc3be7d05ad7c0 parsl.dataflow.memoization INFO: Task 146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99953a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 74 parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845887/1b07f960-0e34-4d29-bf40-2efafa1ec813_characterizeImage_845887_2.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845887/1b07f960-0e34-4d29-bf40-2efafa1ec813_characterizeImage_845887_2.stderr parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 147 submitted for App calibrate, waiting on task 146 parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 147 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 148 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 148 has memoization hash e291a612bb8e610169bc870795eca5dc parsl.dataflow.memoization INFO: Task 148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9995760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 75 parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845937/ee634d28-bb24-424f-8807-5913155a88bb_characterizeImage_845937_2.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845937/ee634d28-bb24-424f-8807-5913155a88bb_characterizeImage_845937_2.stderr parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 149 submitted for App calibrate, waiting on task 148 parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 149 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 150 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 150 has memoization hash 9d9f8f84922b633fc8f11ababe429bd0 parsl.dataflow.memoization INFO: Task 150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9995940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 76 parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845878/d5f15356-f090-4ae5-8e64-7f0360398e16_characterizeImage_845878_2.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845878/d5f15356-f090-4ae5-8e64-7f0360398e16_characterizeImage_845878_2.stderr parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 151 submitted for App calibrate, waiting on task 150 parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 151 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 152 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 152 has memoization hash 87313c5ab2f030d7d5ba3ec97bcefa53 parsl.dataflow.memoization INFO: Task 152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9995b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 77 parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845911/c26ae543-124e-43ed-9770-026445d465de_characterizeImage_845911_2.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845911/c26ae543-124e-43ed-9770-026445d465de_characterizeImage_845911_2.stderr parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 153 submitted for App calibrate, waiting on task 152 parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 153 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 154 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 154 has memoization hash 7859859e4a8b3acfa0e25ffd22bd890e parsl.dataflow.memoization INFO: Task 154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9995ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 78 parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845935/155c48c4-59fe-4597-9ead-47a1755574bb_characterizeImage_845935_2.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845935/155c48c4-59fe-4597-9ead-47a1755574bb_characterizeImage_845935_2.stderr parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 155 submitted for App calibrate, waiting on task 154 parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 155 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 156 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 156 has memoization hash ccc2b86bd1c61d8b970774aae6f9d230 parsl.dataflow.memoization INFO: Task 156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9995d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 79 parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845912/437c7ef3-b918-4ddb-bf93-6a9c10183898_characterizeImage_845912_2.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845912/437c7ef3-b918-4ddb-bf93-6a9c10183898_characterizeImage_845912_2.stderr parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 157 submitted for App calibrate, waiting on task 156 parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 157 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 158 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 158 has memoization hash d89618643baaf0770e9df36d002fbc8f parsl.dataflow.memoization INFO: Task 158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99960c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 80 parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845963/70bff48a-f65e-4dcb-81d1-ac5f7b395120_characterizeImage_845963_2.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845963/70bff48a-f65e-4dcb-81d1-ac5f7b395120_characterizeImage_845963_2.stderr parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 159 submitted for App calibrate, waiting on task 158 parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 159 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 160 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 160 has memoization hash 205471985a59a314bc1f41d58fb64027 parsl.dataflow.memoization INFO: Task 160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99962a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 81 parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845968/64a13c94-f9ce-4b3f-81c7-49ecfa48fb85_characterizeImage_845968_2.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845968/64a13c94-f9ce-4b3f-81c7-49ecfa48fb85_characterizeImage_845968_2.stderr parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 161 submitted for App calibrate, waiting on task 160 parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 161 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 162 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 162 has memoization hash 2f450815e21e44b313d3883b69ad19e3 parsl.dataflow.memoization INFO: Task 162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9996480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 82 parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845903/da3e7f1f-c1f6-4f7b-a2f3-02a28506d989_characterizeImage_845903_2.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845903/da3e7f1f-c1f6-4f7b-a2f3-02a28506d989_characterizeImage_845903_2.stderr parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 163 submitted for App calibrate, waiting on task 162 parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 163 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 164 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 164 has memoization hash cbf9ce955b61128a561db8b181be4d0f parsl.dataflow.memoization INFO: Task 164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9996840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 83 parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845898/28352836-5c66-4e8b-a3bb-a7128f2e6b3c_characterizeImage_845898_2.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845898/28352836-5c66-4e8b-a3bb-a7128f2e6b3c_characterizeImage_845898_2.stderr parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 165 submitted for App calibrate, waiting on task 164 parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 165 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 166 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 166 has memoization hash 6018698b71a085912efb29d8c31fb68b parsl.dataflow.memoization INFO: Task 166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9996660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 84 parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845884/cfcb4712-11f0-43a3-90d5-569ed4c00657_characterizeImage_845884_2.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845884/cfcb4712-11f0-43a3-90d5-569ed4c00657_characterizeImage_845884_2.stderr parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 167 submitted for App calibrate, waiting on task 166 parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 167 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 168 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 168 has memoization hash 84f94359e378d4bd0ce4dbcf2dd93e4a parsl.dataflow.memoization INFO: Task 168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9996a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 85 parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845875/9bc5aca3-17c2-45c9-9565-5995c7db1735_characterizeImage_845875_2.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845875/9bc5aca3-17c2-45c9-9565-5995c7db1735_characterizeImage_845875_2.stderr parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 169 submitted for App calibrate, waiting on task 168 parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 169 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 170 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 170 has memoization hash cf501f8bed69538cab5578a17074a0ee parsl.dataflow.memoization INFO: Task 170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9996c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 170 try 0 launched on executor multi with executor id 86 parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845904/5f371e5b-014c-435d-bf1e-95319d9c08a8_characterizeImage_845904_2.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845904/5f371e5b-014c-435d-bf1e-95319d9c08a8_characterizeImage_845904_2.stderr parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 171 submitted for App calibrate, waiting on task 170 parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 171 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 172 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 172 has memoization hash dbebc46a58a8c7b92a3991cf1867c43b parsl.dataflow.memoization INFO: Task 172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9996de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 87 parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845881/1ef47dee-dced-4ac3-9577-1603c7c7b18a_characterizeImage_845881_2.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845881/1ef47dee-dced-4ac3-9577-1603c7c7b18a_characterizeImage_845881_2.stderr parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 173 submitted for App calibrate, waiting on task 172 parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 173 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 174 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 174 has memoization hash 795e44e2a3bf76947633282bb3375863 parsl.dataflow.memoization INFO: Task 174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99971a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 88 parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845882/dd838f9a-4c46-46e4-a945-5f2a3d305458_characterizeImage_845882_2.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845882/dd838f9a-4c46-46e4-a945-5f2a3d305458_characterizeImage_845882_2.stderr parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 175 submitted for App calibrate, waiting on task 174 parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 175 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 176 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 176 has memoization hash 5f1bee9a3d168108278ed93378da91d3 parsl.dataflow.memoization INFO: Task 176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9996fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 89 parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845872/f877b180-ff8a-4843-b8b3-6f68ec528560_characterizeImage_845872_2.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845872/f877b180-ff8a-4843-b8b3-6f68ec528560_characterizeImage_845872_2.stderr parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 177 submitted for App calibrate, waiting on task 176 parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 177 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 178 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 178 has memoization hash 28643aa54d498a59227ecef6d1e72523 parsl.dataflow.memoization INFO: Task 178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9997380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 90 parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845900/488d80a1-d095-4def-b45a-12fc69abfa32_characterizeImage_845900_2.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845900/488d80a1-d095-4def-b45a-12fc69abfa32_characterizeImage_845900_2.stderr parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 179 submitted for App calibrate, waiting on task 178 parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 179 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 180 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 180 has memoization hash dfd9556d2e7a174dcdd1fbe33721d23e parsl.dataflow.memoization INFO: Task 180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9997560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 91 parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845922/eb971c87-0a6c-4a04-9d00-74a1f966b764_characterizeImage_845922_2.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845922/eb971c87-0a6c-4a04-9d00-74a1f966b764_characterizeImage_845922_2.stderr parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 181 submitted for App calibrate, waiting on task 180 parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 181 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 182 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 182 has memoization hash 83c32e90a036e41a982dc399a7d6f5af parsl.dataflow.memoization INFO: Task 182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9997740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 92 parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845950/6f1a7668-a044-4ff6-b762-1a6e2bfc97f2_characterizeImage_845950_2.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845950/6f1a7668-a044-4ff6-b762-1a6e2bfc97f2_characterizeImage_845950_2.stderr parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 183 submitted for App calibrate, waiting on task 182 parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 183 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 184 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 184 has memoization hash 6be35a8648e1a0dc2874bb5e842da878 parsl.dataflow.memoization INFO: Task 184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9997b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 93 parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845952/2d5dcdc4-3ad9-4a50-9879-e6bf6c638b35_characterizeImage_845952_2.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845952/2d5dcdc4-3ad9-4a50-9879-e6bf6c638b35_characterizeImage_845952_2.stderr parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 185 submitted for App calibrate, waiting on task 184 parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 185 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 186 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 186 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 186 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 186 has memoization hash 785d72514708161c47089ddda2d9876b parsl.dataflow.memoization INFO: Task 186 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9997920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 186 try 0 launched on executor multi with executor id 94 parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845948/d09b2cb0-abb4-4627-a85a-84f4a7ef645a_characterizeImage_845948_2.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845948/d09b2cb0-abb4-4627-a85a-84f4a7ef645a_characterizeImage_845948_2.stderr parsl.dataflow.dflow DEBUG: Task 187 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 187 submitted for App calibrate, waiting on task 186 parsl.dataflow.dflow DEBUG: Task 187 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 187 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 188 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 188 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 188 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 188 has memoization hash ab9ec5bc927e597b0f919b042eb5fe90 parsl.dataflow.memoization INFO: Task 188 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9997ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 188 try 0 launched on executor multi with executor id 95 parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845959/68a938ce-bc8c-4b0b-9e09-71045e95c554_characterizeImage_845959_2.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845959/68a938ce-bc8c-4b0b-9e09-71045e95c554_characterizeImage_845959_2.stderr parsl.dataflow.dflow DEBUG: Task 189 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 189 submitted for App calibrate, waiting on task 188 parsl.dataflow.dflow DEBUG: Task 189 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 189 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 190 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 190 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 190 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 190 has memoization hash b1abbbdcd778eecc9929af6c7c01b512 parsl.dataflow.memoization INFO: Task 190 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99d8040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 190 try 0 launched on executor multi with executor id 96 parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845930/f42fccef-f24c-44d0-83d0-c0b1dc938d0d_characterizeImage_845930_2.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845930/f42fccef-f24c-44d0-83d0-c0b1dc938d0d_characterizeImage_845930_2.stderr parsl.dataflow.dflow DEBUG: Task 191 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 191 submitted for App calibrate, waiting on task 190 parsl.dataflow.dflow DEBUG: Task 191 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 191 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 192 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 192 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 192 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 192 has memoization hash a376d0b9b6e26b25865f60ddcbedf788 parsl.dataflow.memoization INFO: Task 192 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99d80e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 192 try 0 launched on executor multi with executor id 97 parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845914/ebd9c59d-f745-4d03-a038-cf58a534897d_characterizeImage_845914_2.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845914/ebd9c59d-f745-4d03-a038-cf58a534897d_characterizeImage_845914_2.stderr parsl.dataflow.dflow DEBUG: Task 193 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 193 submitted for App calibrate, waiting on task 192 parsl.dataflow.dflow DEBUG: Task 193 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 193 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 194 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 194 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 194 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 194 has memoization hash 44cd58cd5c3776a198d2183f17aba522 parsl.dataflow.memoization INFO: Task 194 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99d84a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 194 try 0 launched on executor multi with executor id 98 parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845974/12299a58-5e4e-409a-9e2e-f61ec6aea500_characterizeImage_845974_2.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845974/12299a58-5e4e-409a-9e2e-f61ec6aea500_characterizeImage_845974_2.stderr parsl.dataflow.dflow DEBUG: Task 195 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 195 submitted for App calibrate, waiting on task 194 parsl.dataflow.dflow DEBUG: Task 195 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 195 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 196 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 196 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 196 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 196 has memoization hash 734c3b65efd04261386a77579ffb4721 parsl.dataflow.memoization INFO: Task 196 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99d82c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 196 try 0 launched on executor multi with executor id 99 parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845932/d3fd6454-1902-4c2b-84e1-0ae86cd08049_characterizeImage_845932_2.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845932/d3fd6454-1902-4c2b-84e1-0ae86cd08049_characterizeImage_845932_2.stderr parsl.dataflow.dflow DEBUG: Task 197 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 197 submitted for App calibrate, waiting on task 196 parsl.dataflow.dflow DEBUG: Task 197 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 197 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 198 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 198 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 198 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 198 has memoization hash 5932264ca88347b2fcaaa05a168af8dd parsl.dataflow.memoization INFO: Task 198 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99d8680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 198 try 0 launched on executor multi with executor id 100 parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845905/451fa2d1-c592-4332-b492-397ff2639846_characterizeImage_845905_2.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845905/451fa2d1-c592-4332-b492-397ff2639846_characterizeImage_845905_2.stderr parsl.dataflow.dflow DEBUG: Task 199 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 199 submitted for App calibrate, waiting on task 198 parsl.dataflow.dflow DEBUG: Task 199 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 199 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 200 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 200 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 200 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 200 has memoization hash b95ed119de64950641922d71b294bf1e parsl.dataflow.memoization INFO: Task 200 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99d8860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 200 try 0 launched on executor multi with executor id 101 parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845966/a7a8e70c-92d2-46fa-9a07-b1cec0342def_characterizeImage_845966_2.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845966/a7a8e70c-92d2-46fa-9a07-b1cec0342def_characterizeImage_845966_2.stderr parsl.dataflow.dflow DEBUG: Task 201 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 201 submitted for App calibrate, waiting on task 200 parsl.dataflow.dflow DEBUG: Task 201 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 201 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 202 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 202 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 202 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 202 has memoization hash 2c2197ac36febbc47c3e875fc012069d parsl.dataflow.memoization INFO: Task 202 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99d8a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 202 try 0 launched on executor multi with executor id 102 parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845972/11cfd015-e4eb-41aa-a464-f15678e8e693_characterizeImage_845972_2.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845972/11cfd015-e4eb-41aa-a464-f15678e8e693_characterizeImage_845972_2.stderr parsl.dataflow.dflow DEBUG: Task 203 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 203 submitted for App calibrate, waiting on task 202 parsl.dataflow.dflow DEBUG: Task 203 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 203 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 204 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 204 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 204 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 204 has memoization hash b369fb9f690069e88e8d95a11e339689 parsl.dataflow.memoization INFO: Task 204 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99d8e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 204 try 0 launched on executor multi with executor id 103 parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845877/a83d1524-ba7c-40ed-a158-6d1244bf3462_characterizeImage_845877_2.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845877/a83d1524-ba7c-40ed-a158-6d1244bf3462_characterizeImage_845877_2.stderr parsl.dataflow.dflow DEBUG: Task 205 will be sent to executor multi 2024-03-13 00:15:11 proc_lsst.multi:146 [INFO] found job 10228 in provider local proc_lsst.multi INFO: found job 10228 in provider local parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 205 submitted for App calibrate, waiting on task 204 parsl.dataflow.dflow DEBUG: Task 205 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 205 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 206 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 206 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 206 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 206 has memoization hash 414b1d50f04b6239e5666b74acba768d parsl.dataflow.memoization INFO: Task 206 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99dbd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 206 try 0 launched on executor multi with executor id 104 parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845973/ebbf665d-168b-4cef-b190-f403cefe8c46_characterizeImage_845973_2.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845973/ebbf665d-168b-4cef-b190-f403cefe8c46_characterizeImage_845973_2.stderr parsl.dataflow.dflow DEBUG: Task 207 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 207 submitted for App calibrate, waiting on task 206 parsl.dataflow.dflow DEBUG: Task 207 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 207 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 104 parsl.dataflow.strategy DEBUG: Executor multi has 104 active tasks, 1/0 running/pending blocks, and 0 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 25 more blocks parsl.executors.status_handling INFO: Scaling out by 25 blocks parsl.executors.status_handling INFO: Allocated block ID 1 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1 2024-03-13 00:15:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 2024-03-13 00:15:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 00:15:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 00:15:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 00:15:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 00:15:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-13 00:15:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 00:15:11 proc_lsst.multi:170 [INFO] [multi] job_id 17006354 proc_lsst.multi INFO: [multi] job_id 17006354 2024-03-13 00:15:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 00:15:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:11 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006354 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006354 parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 17006354 parsl.executors.status_handling INFO: Allocated block ID 2 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2 2024-03-13 00:15:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 2024-03-13 00:15:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 00:15:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 00:15:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 00:15:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 00:15:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 00:15:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 00:15:12 proc_lsst.multi:170 [INFO] [multi] job_id 17006355 proc_lsst.multi INFO: [multi] job_id 17006355 2024-03-13 00:15:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-13 00:15:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:12 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006355 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006355 parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 17006355 parsl.executors.status_handling INFO: Allocated block ID 3 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3 2024-03-13 00:15:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 2024-03-13 00:15:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 00:15:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 00:15:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 00:15:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 00:15:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-13 00:15:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 00:15:12 proc_lsst.multi:170 [INFO] [multi] job_id 17006356 proc_lsst.multi INFO: [multi] job_id 17006356 2024-03-13 00:15:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-13 00:15:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:12 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006356 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006356 parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 17006356 parsl.executors.status_handling INFO: Allocated block ID 4 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4 2024-03-13 00:15:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 2024-03-13 00:15:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 00:15:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 00:15:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 00:15:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 00:15:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-13 00:15:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 00:15:13 proc_lsst.multi:170 [INFO] [multi] job_id 17006357 proc_lsst.multi INFO: [multi] job_id 17006357 2024-03-13 00:15:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-13 00:15:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:13 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006357 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006357 parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 17006357 parsl.executors.status_handling INFO: Allocated block ID 5 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5 2024-03-13 00:15:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 2024-03-13 00:15:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 00:15:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 00:15:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 00:15:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 00:15:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-13 00:15:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 00:15:13 proc_lsst.multi:170 [INFO] [multi] job_id 17006358 proc_lsst.multi INFO: [multi] job_id 17006358 2024-03-13 00:15:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 00:15:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:13 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006358 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006358 parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 17006358 parsl.executors.status_handling INFO: Allocated block ID 6 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6 2024-03-13 00:15:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 2024-03-13 00:15:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 00:15:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 00:15:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 00:15:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 00:15:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 00:15:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 00:15:13 proc_lsst.multi:170 [INFO] [multi] job_id 17006359 proc_lsst.multi INFO: [multi] job_id 17006359 2024-03-13 00:15:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-13 00:15:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:13 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006359 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006359 parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 17006359 parsl.executors.status_handling INFO: Allocated block ID 7 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7 2024-03-13 00:15:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 2024-03-13 00:15:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 00:15:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 00:15:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 00:15:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 00:15:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-13 00:15:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 00:15:14 proc_lsst.multi:170 [INFO] [multi] job_id 17006360 proc_lsst.multi INFO: [multi] job_id 17006360 2024-03-13 00:15:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-13 00:15:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:14 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006360 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006360 parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 17006360 parsl.executors.status_handling INFO: Allocated block ID 8 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8 2024-03-13 00:15:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 2024-03-13 00:15:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 00:15:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 00:15:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 00:15:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 00:15:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-13 00:15:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 00:15:14 proc_lsst.multi:170 [INFO] [multi] job_id 17006361 proc_lsst.multi INFO: [multi] job_id 17006361 2024-03-13 00:15:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-13 00:15:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:14 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006361 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006361 parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 17006361 parsl.executors.status_handling INFO: Allocated block ID 9 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9 2024-03-13 00:15:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 2024-03-13 00:15:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 00:15:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 00:15:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 00:15:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 00:15:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-13 00:15:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 00:15:15 proc_lsst.multi:170 [INFO] [multi] job_id 17006362 proc_lsst.multi INFO: [multi] job_id 17006362 2024-03-13 00:15:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-13 00:15:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:15 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006362 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006362 parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 17006362 parsl.executors.status_handling INFO: Allocated block ID 10 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10 2024-03-13 00:15:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 2024-03-13 00:15:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 00:15:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 00:15:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 00:15:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 00:15:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-13 00:15:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 00:15:15 proc_lsst.multi:170 [INFO] [multi] job_id 17006363 proc_lsst.multi INFO: [multi] job_id 17006363 2024-03-13 00:15:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-13 00:15:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:15 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006363 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006363 parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 17006363 parsl.executors.status_handling INFO: Allocated block ID 11 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11 2024-03-13 00:15:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 2024-03-13 00:15:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 00:15:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 00:15:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 00:15:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 00:15:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-13 00:15:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 00:15:15 proc_lsst.multi:170 [INFO] [multi] job_id 17006364 proc_lsst.multi INFO: [multi] job_id 17006364 2024-03-13 00:15:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-13 00:15:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:15 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006364 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006364 parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 17006364 parsl.executors.status_handling INFO: Allocated block ID 12 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12 2024-03-13 00:15:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 2024-03-13 00:15:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 00:15:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 00:15:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 00:15:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 00:15:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-13 00:15:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 00:15:16 proc_lsst.multi:170 [INFO] [multi] job_id 17006365 proc_lsst.multi INFO: [multi] job_id 17006365 2024-03-13 00:15:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-13 00:15:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:16 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006365 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006365 parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 17006365 parsl.executors.status_handling INFO: Allocated block ID 13 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13 2024-03-13 00:15:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 2024-03-13 00:15:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 00:15:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 00:15:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 00:15:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 00:15:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-13 00:15:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 00:15:16 proc_lsst.multi:170 [INFO] [multi] job_id 17006366 proc_lsst.multi INFO: [multi] job_id 17006366 2024-03-13 00:15:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-13 00:15:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:16 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006366 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006366 parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 17006366 parsl.executors.status_handling INFO: Allocated block ID 14 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14 2024-03-13 00:15:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 2024-03-13 00:15:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 00:15:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 00:15:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 00:15:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 00:15:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-13 00:15:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 00:15:17 proc_lsst.multi:170 [INFO] [multi] job_id 17006367 proc_lsst.multi INFO: [multi] job_id 17006367 2024-03-13 00:15:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-13 00:15:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:17 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006367 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006367 parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 17006367 parsl.executors.status_handling INFO: Allocated block ID 15 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15 2024-03-13 00:15:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 2024-03-13 00:15:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 00:15:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 00:15:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 00:15:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 00:15:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-13 00:15:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 00:15:17 proc_lsst.multi:170 [INFO] [multi] job_id 17006368 proc_lsst.multi INFO: [multi] job_id 17006368 2024-03-13 00:15:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-13 00:15:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:17 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006368 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006368 parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 17006368 parsl.executors.status_handling INFO: Allocated block ID 16 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16 2024-03-13 00:15:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 2024-03-13 00:15:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 00:15:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 00:15:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 00:15:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 00:15:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-13 00:15:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 00:15:18 proc_lsst.multi:170 [INFO] [multi] job_id 17006369 proc_lsst.multi INFO: [multi] job_id 17006369 2024-03-13 00:15:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-13 00:15:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:18 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006369 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006369 parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 17006369 parsl.executors.status_handling INFO: Allocated block ID 17 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17 2024-03-13 00:15:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 2024-03-13 00:15:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 00:15:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 00:15:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 00:15:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 00:15:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-13 00:15:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 00:15:18 proc_lsst.multi:170 [INFO] [multi] job_id 17006370 proc_lsst.multi INFO: [multi] job_id 17006370 2024-03-13 00:15:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-13 00:15:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:18 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006370 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006370 parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 17006370 parsl.executors.status_handling INFO: Allocated block ID 18 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18 2024-03-13 00:15:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 2024-03-13 00:15:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 00:15:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 00:15:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 00:15:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 00:15:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-13 00:15:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 00:15:19 proc_lsst.multi:170 [INFO] [multi] job_id 17006371 proc_lsst.multi INFO: [multi] job_id 17006371 2024-03-13 00:15:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-13 00:15:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:19 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006371 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006371 parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 17006371 parsl.executors.status_handling INFO: Allocated block ID 19 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19 2024-03-13 00:15:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 2024-03-13 00:15:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 00:15:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 00:15:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 00:15:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 00:15:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-13 00:15:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 00:15:19 proc_lsst.multi:170 [INFO] [multi] job_id 17006372 proc_lsst.multi INFO: [multi] job_id 17006372 2024-03-13 00:15:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-13 00:15:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:19 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006372 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006372 parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 17006372 parsl.executors.status_handling INFO: Allocated block ID 20 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20 2024-03-13 00:15:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 2024-03-13 00:15:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 00:15:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 00:15:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 00:15:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 00:15:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-13 00:15:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 00:15:20 proc_lsst.multi:170 [INFO] [multi] job_id 17006373 proc_lsst.multi INFO: [multi] job_id 17006373 2024-03-13 00:15:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 00:15:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:20 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006373 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006373 parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 17006373 parsl.executors.status_handling INFO: Allocated block ID 21 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21 2024-03-13 00:15:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 2024-03-13 00:15:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 00:15:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 00:15:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 00:15:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 00:15:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 00:15:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 00:15:20 proc_lsst.multi:170 [INFO] [multi] job_id 17006374 proc_lsst.multi INFO: [multi] job_id 17006374 2024-03-13 00:15:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-13 00:15:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:20 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006374 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006374 parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 17006374 parsl.executors.status_handling INFO: Allocated block ID 22 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22 2024-03-13 00:15:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 2024-03-13 00:15:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 00:15:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 00:15:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 00:15:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 00:15:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-13 00:15:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 00:15:20 proc_lsst.multi:170 [INFO] [multi] job_id 17006375 proc_lsst.multi INFO: [multi] job_id 17006375 2024-03-13 00:15:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-13 00:15:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:20 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006375 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006375 parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 17006375 parsl.executors.status_handling INFO: Allocated block ID 23 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23 2024-03-13 00:15:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 2024-03-13 00:15:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 00:15:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 00:15:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 00:15:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 00:15:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-13 00:15:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 00:15:21 proc_lsst.multi:170 [INFO] [multi] job_id 17006376 proc_lsst.multi INFO: [multi] job_id 17006376 2024-03-13 00:15:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-13 00:15:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:21 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006376 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006376 parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 17006376 parsl.executors.status_handling INFO: Allocated block ID 24 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24 2024-03-13 00:15:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 2024-03-13 00:15:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 00:15:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 00:15:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 00:15:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 00:15:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-13 00:15:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 2 try 0 failed parsl.dataflow.dflow INFO: Task 2 marked for retry parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845969/e5463690-4c90-49b8-bf95-413e46148a08_characterizeImage_845969_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845969/e5463690-4c90-49b8-bf95-413e46148a08_characterizeImage_845969_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2 has memoization hash 74965c9fb0108800374f15fc3f8aad54 parsl.dataflow.memoization INFO: Task 2 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9938cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2 try 1 launched on executor multi with executor id 105 parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845969/e5463690-4c90-49b8-bf95-413e46148a08_characterizeImage_845969_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845969/e5463690-4c90-49b8-bf95-413e46148a08_characterizeImage_845969_2.stderr 2024-03-13 00:15:21 proc_lsst.multi:170 [INFO] [multi] job_id 17006377 proc_lsst.multi INFO: [multi] job_id 17006377 2024-03-13 00:15:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-13 00:15:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:21 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006377 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006377 parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 17006377 parsl.executors.status_handling INFO: Allocated block ID 25 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25 2024-03-13 00:15:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 2024-03-13 00:15:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 00:15:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 00:15:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 00:15:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 00:15:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-13 00:15:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,169.254.95.120,10.64.65.9,10.64.129.9,n3009,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54770 --result_port=54664 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 0 try 0 failed parsl.dataflow.dflow INFO: Task 0 marked for retry parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845889/47baa340-e987-4003-a9f7-a3eed82a19d4_characterizeImage_845889_2.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845889/47baa340-e987-4003-a9f7-a3eed82a19d4_characterizeImage_845889_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 0 has memoization hash dfcbb6858bd62a02b1c647f383fac018 parsl.dataflow.memoization INFO: Task 0 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9938540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 0 try 1 launched on executor multi with executor id 106 parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845889/47baa340-e987-4003-a9f7-a3eed82a19d4_characterizeImage_845889_2.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845889/47baa340-e987-4003-a9f7-a3eed82a19d4_characterizeImage_845889_2.stderr 2024-03-13 00:15:22 proc_lsst.multi:170 [INFO] [multi] job_id 17006378 proc_lsst.multi INFO: [multi] job_id 17006378 2024-03-13 00:15:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-13 00:15:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 00:15:22 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006378 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006378 parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 17006378 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23218217990800 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 26, active_tasks = 104 parsl.dataflow.strategy DEBUG: Executor multi has 104 active tasks, 1/25 running/pending blocks, and 16 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23218217990800 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 4 try 0 failed parsl.dataflow.dflow INFO: Task 4 marked for retry parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845923/307bb778-9554-4cf5-87a7-59a661125856_characterizeImage_845923_2.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845923/307bb778-9554-4cf5-87a7-59a661125856_characterizeImage_845923_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash 9d87fd5e64d591d853b837f65776411a parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9938fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4 try 1 launched on executor multi with executor id 107 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845923/307bb778-9554-4cf5-87a7-59a661125856_characterizeImage_845923_2.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845923/307bb778-9554-4cf5-87a7-59a661125856_characterizeImage_845923_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 6 try 0 failed parsl.dataflow.dflow INFO: Task 6 marked for retry parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845917/6af4a453-83c9-42cf-b400-c9684629b977_characterizeImage_845917_2.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845917/6af4a453-83c9-42cf-b400-c9684629b977_characterizeImage_845917_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 6 has memoization hash 88e67c74e0c27846fbc6159ad3abc379 parsl.dataflow.memoization INFO: Task 6 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9939080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 6 try 1 launched on executor multi with executor id 108 parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845917/6af4a453-83c9-42cf-b400-c9684629b977_characterizeImage_845917_2.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845917/6af4a453-83c9-42cf-b400-c9684629b977_characterizeImage_845917_2.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 26, active_tasks = 104 parsl.dataflow.strategy DEBUG: Executor multi has 104 active tasks, 1/25 running/pending blocks, and 22 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23218217990800 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 32 try 0 failed parsl.dataflow.dflow INFO: Task 32 marked for retry parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845909/ac020887-2a2a-4cd3-973d-f2cf3d0ca195_characterizeImage_845909_2.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845909/ac020887-2a2a-4cd3-973d-f2cf3d0ca195_characterizeImage_845909_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 32 has memoization hash 469fb8ae935bab5e16cecb86e5f5d753 parsl.dataflow.memoization INFO: Task 32 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de993b2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 32 try 1 launched on executor multi with executor id 109 parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845909/ac020887-2a2a-4cd3-973d-f2cf3d0ca195_characterizeImage_845909_2.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845909/ac020887-2a2a-4cd3-973d-f2cf3d0ca195_characterizeImage_845909_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 34 try 0 failed parsl.dataflow.dflow INFO: Task 34 marked for retry parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845938/04e8f597-23e1-4218-883d-b3fc7d9b8399_characterizeImage_845938_2.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845938/04e8f597-23e1-4218-883d-b3fc7d9b8399_characterizeImage_845938_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash c01c083e32fcf85f4e300591a01b3000 parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de993af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 1 launched on executor multi with executor id 110 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845938/04e8f597-23e1-4218-883d-b3fc7d9b8399_characterizeImage_845938_2.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845938/04e8f597-23e1-4218-883d-b3fc7d9b8399_characterizeImage_845938_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 12 try 0 failed parsl.dataflow.dflow INFO: Task 12 marked for retry parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845925/932d1e2c-a64f-4d9f-bf21-e4e5d4cff2fc_characterizeImage_845925_2.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845925/932d1e2c-a64f-4d9f-bf21-e4e5d4cff2fc_characterizeImage_845925_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 12 has memoization hash 98ac88c7618fcf1413ef0530b9ef8734 parsl.dataflow.memoization INFO: Task 12 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99396c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 12 try 1 launched on executor multi with executor id 111 parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845925/932d1e2c-a64f-4d9f-bf21-e4e5d4cff2fc_characterizeImage_845925_2.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845925/932d1e2c-a64f-4d9f-bf21-e4e5d4cff2fc_characterizeImage_845925_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 8 try 0 failed parsl.dataflow.dflow INFO: Task 8 marked for retry parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845879/9444aec8-7f5a-4bb7-abb6-e19832d51831_characterizeImage_845879_2.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845879/9444aec8-7f5a-4bb7-abb6-e19832d51831_characterizeImage_845879_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 8 has memoization hash 0a8648caf6226ed5ca02cccbe0a7329a parsl.dataflow.memoization INFO: Task 8 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9939120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 8 try 1 launched on executor multi with executor id 112 parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845879/9444aec8-7f5a-4bb7-abb6-e19832d51831_characterizeImage_845879_2.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845879/9444aec8-7f5a-4bb7-abb6-e19832d51831_characterizeImage_845879_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 14 try 0 failed parsl.dataflow.dflow INFO: Task 14 marked for retry parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845876/b79d0412-b23f-4fc3-9028-4c3becfc27d5_characterizeImage_845876_2.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845876/b79d0412-b23f-4fc3-9028-4c3becfc27d5_characterizeImage_845876_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 14 has memoization hash 3159d685790ff3c487a08c113df38067 parsl.dataflow.memoization INFO: Task 14 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9939940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 14 try 1 launched on executor multi with executor id 113 parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845876/b79d0412-b23f-4fc3-9028-4c3becfc27d5_characterizeImage_845876_2.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845876/b79d0412-b23f-4fc3-9028-4c3becfc27d5_characterizeImage_845876_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 10 try 0 failed parsl.dataflow.dflow INFO: Task 10 marked for retry parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845956/9dd128fe-62fa-4ea8-859e-c6327748c2b1_characterizeImage_845956_2.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845956/9dd128fe-62fa-4ea8-859e-c6327748c2b1_characterizeImage_845956_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash 0e239dd7fa281aaaa938ba9801fea216 parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9939300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 10 try 1 launched on executor multi with executor id 114 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845956/9dd128fe-62fa-4ea8-859e-c6327748c2b1_characterizeImage_845956_2.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845956/9dd128fe-62fa-4ea8-859e-c6327748c2b1_characterizeImage_845956_2.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 26, active_tasks = 104 parsl.dataflow.strategy DEBUG: Executor multi has 104 active tasks, 1/25 running/pending blocks, and 22 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23218217990800 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 48 try 0 failed parsl.dataflow.dflow INFO: Task 48 marked for retry parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845891/cd95ec31-d06c-4bdb-a6c3-642fc6d57370_characterizeImage_845891_2.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845891/cd95ec31-d06c-4bdb-a6c3-642fc6d57370_characterizeImage_845891_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 48 has memoization hash 017a71140c924a8d1120a2a44be86333 parsl.dataflow.memoization INFO: Task 48 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de993b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 48 try 1 launched on executor multi with executor id 115 parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845891/cd95ec31-d06c-4bdb-a6c3-642fc6d57370_characterizeImage_845891_2.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845891/cd95ec31-d06c-4bdb-a6c3-642fc6d57370_characterizeImage_845891_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 50 try 0 failed parsl.dataflow.dflow INFO: Task 50 marked for retry parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845953/2bd41c30-2e53-4ce5-9fa8-8eae6750494d_characterizeImage_845953_2.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845953/2bd41c30-2e53-4ce5-9fa8-8eae6750494d_characterizeImage_845953_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 50 has memoization hash d04a3d9d8240c79d5986e53c79af23f9 parsl.dataflow.memoization INFO: Task 50 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de993bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 50 try 1 launched on executor multi with executor id 116 parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845953/2bd41c30-2e53-4ce5-9fa8-8eae6750494d_characterizeImage_845953_2.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845953/2bd41c30-2e53-4ce5-9fa8-8eae6750494d_characterizeImage_845953_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 42 try 0 failed parsl.dataflow.dflow INFO: Task 42 marked for retry parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845916/797024ed-829b-4ed2-9eaf-db520235adbf_characterizeImage_845916_2.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845916/797024ed-829b-4ed2-9eaf-db520235adbf_characterizeImage_845916_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 42 has memoization hash 7ba2785931ac668f5aa3083a3524de72 parsl.dataflow.memoization INFO: Task 42 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de993a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 42 try 1 launched on executor multi with executor id 117 parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845916/797024ed-829b-4ed2-9eaf-db520235adbf_characterizeImage_845916_2.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845916/797024ed-829b-4ed2-9eaf-db520235adbf_characterizeImage_845916_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 40 try 0 failed parsl.dataflow.dflow INFO: Task 40 marked for retry parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845926/5a3176b1-cf38-4e9b-9585-03024c203f7e_characterizeImage_845926_2.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845926/5a3176b1-cf38-4e9b-9585-03024c203f7e_characterizeImage_845926_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash fed050588955d9cdeb4e4959aef78ddf parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de993ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 40 try 1 launched on executor multi with executor id 118 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845926/5a3176b1-cf38-4e9b-9585-03024c203f7e_characterizeImage_845926_2.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845926/5a3176b1-cf38-4e9b-9585-03024c203f7e_characterizeImage_845926_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 44 try 0 failed parsl.dataflow.dflow INFO: Task 44 marked for retry parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845954/eaf6cc58-5cb9-4477-924a-e33e022dcc7f_characterizeImage_845954_2.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845954/eaf6cc58-5cb9-4477-924a-e33e022dcc7f_characterizeImage_845954_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 44 has memoization hash 070ff849bee17ce25704e41606eee33c parsl.dataflow.memoization INFO: Task 44 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de993a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 44 try 1 launched on executor multi with executor id 119 parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845954/eaf6cc58-5cb9-4477-924a-e33e022dcc7f_characterizeImage_845954_2.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845954/eaf6cc58-5cb9-4477-924a-e33e022dcc7f_characterizeImage_845954_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 38 try 0 failed parsl.dataflow.dflow INFO: Task 38 marked for retry parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845920/38b5da06-64ee-4e25-965b-c964097db327_characterizeImage_845920_2.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845920/38b5da06-64ee-4e25-965b-c964097db327_characterizeImage_845920_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 38 has memoization hash f64e69ad7b5d5c9b4323b3d25b5f347f parsl.dataflow.memoization INFO: Task 38 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de993ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 38 try 1 launched on executor multi with executor id 120 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845920/38b5da06-64ee-4e25-965b-c964097db327_characterizeImage_845920_2.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845920/38b5da06-64ee-4e25-965b-c964097db327_characterizeImage_845920_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 46 try 0 failed parsl.dataflow.dflow INFO: Task 46 marked for retry parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845886/9251d4db-ca14-412f-92e9-66fdd14198d5_characterizeImage_845886_2.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845886/9251d4db-ca14-412f-92e9-66fdd14198d5_characterizeImage_845886_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 46 has memoization hash e890f1b1d6866bc6c740aa0eec7e066e parsl.dataflow.memoization INFO: Task 46 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de993a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 46 try 1 launched on executor multi with executor id 121 parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845886/9251d4db-ca14-412f-92e9-66fdd14198d5_characterizeImage_845886_2.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845886/9251d4db-ca14-412f-92e9-66fdd14198d5_characterizeImage_845886_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 36 try 0 failed parsl.dataflow.dflow INFO: Task 36 marked for retry parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845888/6662b1fa-0e7e-445c-94b7-c3e176b4f763_characterizeImage_845888_2.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845888/6662b1fa-0e7e-445c-94b7-c3e176b4f763_characterizeImage_845888_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 36 has memoization hash e5f44513b16c88757ba89a5eba595551 parsl.dataflow.memoization INFO: Task 36 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de993b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 36 try 1 launched on executor multi with executor id 122 parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845888/6662b1fa-0e7e-445c-94b7-c3e176b4f763_characterizeImage_845888_2.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845888/6662b1fa-0e7e-445c-94b7-c3e176b4f763_characterizeImage_845888_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 24 try 0 failed parsl.dataflow.dflow INFO: Task 24 marked for retry parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845929/1a960c36-216f-4b57-b1f9-5a6150aadd7a_characterizeImage_845929_2.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845929/1a960c36-216f-4b57-b1f9-5a6150aadd7a_characterizeImage_845929_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 24 has memoization hash 18e2ca751e16fca96e811f0c522e28ba parsl.dataflow.memoization INFO: Task 24 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de993a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 24 try 1 launched on executor multi with executor id 123 parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845929/1a960c36-216f-4b57-b1f9-5a6150aadd7a_characterizeImage_845929_2.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845929/1a960c36-216f-4b57-b1f9-5a6150aadd7a_characterizeImage_845929_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 30 try 0 failed parsl.dataflow.dflow INFO: Task 30 marked for retry parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845906/6892a379-1d64-46ab-9ec4-ce60e4af68dc_characterizeImage_845906_2.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845906/6892a379-1d64-46ab-9ec4-ce60e4af68dc_characterizeImage_845906_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 30 has memoization hash ba2ae0daa7158fc9112c89a875f03037 parsl.dataflow.memoization INFO: Task 30 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de993b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 30 try 1 launched on executor multi with executor id 124 parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845906/6892a379-1d64-46ab-9ec4-ce60e4af68dc_characterizeImage_845906_2.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845906/6892a379-1d64-46ab-9ec4-ce60e4af68dc_characterizeImage_845906_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 18 try 0 failed parsl.dataflow.dflow INFO: Task 18 marked for retry parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845951/4cab1fe7-79b5-4f3d-8f17-0b8b0e1dea22_characterizeImage_845951_2.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845951/4cab1fe7-79b5-4f3d-8f17-0b8b0e1dea22_characterizeImage_845951_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 18 has memoization hash e996bdc0ebe1e7c59b4fc3cd6b3bd534 parsl.dataflow.memoization INFO: Task 18 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9939c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 18 try 1 launched on executor multi with executor id 125 parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845951/4cab1fe7-79b5-4f3d-8f17-0b8b0e1dea22_characterizeImage_845951_2.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845951/4cab1fe7-79b5-4f3d-8f17-0b8b0e1dea22_characterizeImage_845951_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 20 try 0 failed parsl.dataflow.dflow INFO: Task 20 marked for retry parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845949/f3355e5d-55be-4f4a-883f-ade688be4595_characterizeImage_845949_2.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845949/f3355e5d-55be-4f4a-883f-ade688be4595_characterizeImage_845949_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 20 has memoization hash 77ae5deac8c2a3a9f7fac6e9a5f42fbf parsl.dataflow.memoization INFO: Task 20 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9939e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 20 try 1 launched on executor multi with executor id 126 parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845949/f3355e5d-55be-4f4a-883f-ade688be4595_characterizeImage_845949_2.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845949/f3355e5d-55be-4f4a-883f-ade688be4595_characterizeImage_845949_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 26 try 0 failed parsl.dataflow.dflow INFO: Task 26 marked for retry parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845924/ec22148e-7dcf-487c-85ce-66bade631426_characterizeImage_845924_2.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845924/ec22148e-7dcf-487c-85ce-66bade631426_characterizeImage_845924_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 26 has memoization hash 048d028a81ac065b28de0c859d43adb6 parsl.dataflow.memoization INFO: Task 26 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de993b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 26 try 1 launched on executor multi with executor id 127 parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845924/ec22148e-7dcf-487c-85ce-66bade631426_characterizeImage_845924_2.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845924/ec22148e-7dcf-487c-85ce-66bade631426_characterizeImage_845924_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 22 try 0 failed parsl.dataflow.dflow INFO: Task 22 marked for retry parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845962/9a3ed4f5-fec4-4bf8-80ff-9071e87fee48_characterizeImage_845962_2.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845962/9a3ed4f5-fec4-4bf8-80ff-9071e87fee48_characterizeImage_845962_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash fbcfea2073e27e6eda147bae35d01516 parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9939da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 1 launched on executor multi with executor id 128 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845962/9a3ed4f5-fec4-4bf8-80ff-9071e87fee48_characterizeImage_845962_2.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845962/9a3ed4f5-fec4-4bf8-80ff-9071e87fee48_characterizeImage_845962_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 16 try 0 failed parsl.dataflow.dflow INFO: Task 16 marked for retry parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845955/680a196a-21f7-494d-bc9c-ca5d4d30c999_characterizeImage_845955_2.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845955/680a196a-21f7-494d-bc9c-ca5d4d30c999_characterizeImage_845955_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash cf4b320082b76b1e2fe672aab84cc0ec parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9938860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 16 try 1 launched on executor multi with executor id 129 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845955/680a196a-21f7-494d-bc9c-ca5d4d30c999_characterizeImage_845955_2.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845955/680a196a-21f7-494d-bc9c-ca5d4d30c999_characterizeImage_845955_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 28 try 0 failed parsl.dataflow.dflow INFO: Task 28 marked for retry parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845895/c2b6501b-7148-4734-8d01-d113a76b8478_characterizeImage_845895_2.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845895/c2b6501b-7148-4734-8d01-d113a76b8478_characterizeImage_845895_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash 719d7be68431e19edca9463945dbee24 parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de993b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 1 launched on executor multi with executor id 130 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845895/c2b6501b-7148-4734-8d01-d113a76b8478_characterizeImage_845895_2.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845895/c2b6501b-7148-4734-8d01-d113a76b8478_characterizeImage_845895_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 52 try 0 failed parsl.dataflow.dflow INFO: Task 52 marked for retry parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845893/52c3beab-ecdd-4b0e-9db2-f6b12d2eb801_characterizeImage_845893_2.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845893/52c3beab-ecdd-4b0e-9db2-f6b12d2eb801_characterizeImage_845893_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash 58e47534fd0397325e9e00c6d86effed parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de993bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 1 launched on executor multi with executor id 131 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845893/52c3beab-ecdd-4b0e-9db2-f6b12d2eb801_characterizeImage_845893_2.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845893/52c3beab-ecdd-4b0e-9db2-f6b12d2eb801_characterizeImage_845893_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 54 try 0 failed parsl.dataflow.dflow INFO: Task 54 marked for retry parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845927/86c93158-c0d4-4a62-8934-37cd77914d58_characterizeImage_845927_2.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845927/86c93158-c0d4-4a62-8934-37cd77914d58_characterizeImage_845927_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 54 has memoization hash b5008116ffc61a6f3007ebe16fda163e parsl.dataflow.memoization INFO: Task 54 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99680e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 54 try 1 launched on executor multi with executor id 132 parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845927/86c93158-c0d4-4a62-8934-37cd77914d58_characterizeImage_845927_2.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845927/86c93158-c0d4-4a62-8934-37cd77914d58_characterizeImage_845927_2.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 26, active_tasks = 104 parsl.dataflow.strategy DEBUG: Executor multi has 104 active tasks, 1/25 running/pending blocks, and 22 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23218217990800 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 56 try 0 failed parsl.dataflow.dflow INFO: Task 56 marked for retry parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845928/ea42931b-5e2c-48e0-908f-ad9a56071956_characterizeImage_845928_2.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845928/ea42931b-5e2c-48e0-908f-ad9a56071956_characterizeImage_845928_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 56 has memoization hash f1f634f78f5dbf3f26226408456d4069 parsl.dataflow.memoization INFO: Task 56 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9968180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 56 try 1 launched on executor multi with executor id 133 parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845928/ea42931b-5e2c-48e0-908f-ad9a56071956_characterizeImage_845928_2.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845928/ea42931b-5e2c-48e0-908f-ad9a56071956_characterizeImage_845928_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 58 try 0 failed parsl.dataflow.dflow INFO: Task 58 marked for retry parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845892/f31b5b11-e4a4-4d7d-a8a9-d6d25d84966c_characterizeImage_845892_2.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845892/f31b5b11-e4a4-4d7d-a8a9-d6d25d84966c_characterizeImage_845892_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 58 has memoization hash 16680ccd761e9f2b2599a7273b360be1 parsl.dataflow.memoization INFO: Task 58 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99682c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 58 try 1 launched on executor multi with executor id 134 parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845892/f31b5b11-e4a4-4d7d-a8a9-d6d25d84966c_characterizeImage_845892_2.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845892/f31b5b11-e4a4-4d7d-a8a9-d6d25d84966c_characterizeImage_845892_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 62 try 0 failed parsl.dataflow.dflow INFO: Task 62 marked for retry parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845939/b78c437e-0357-4a72-b7e7-e92ce8ff32e7_characterizeImage_845939_2.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845939/b78c437e-0357-4a72-b7e7-e92ce8ff32e7_characterizeImage_845939_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 62 has memoization hash d8a281c1b3d9f765dac77891d60e22e5 parsl.dataflow.memoization INFO: Task 62 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9968680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 62 try 1 launched on executor multi with executor id 135 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845939/b78c437e-0357-4a72-b7e7-e92ce8ff32e7_characterizeImage_845939_2.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845939/b78c437e-0357-4a72-b7e7-e92ce8ff32e7_characterizeImage_845939_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 60 try 0 failed parsl.dataflow.dflow INFO: Task 60 marked for retry parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845913/024ff1de-e31d-4139-b3c5-b73c055d82ca_characterizeImage_845913_2.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845913/024ff1de-e31d-4139-b3c5-b73c055d82ca_characterizeImage_845913_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 60 has memoization hash 251cbc3ea7dee02d57b87bb8c701f3c7 parsl.dataflow.memoization INFO: Task 60 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99684a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 60 try 1 launched on executor multi with executor id 136 parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845913/024ff1de-e31d-4139-b3c5-b73c055d82ca_characterizeImage_845913_2.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845913/024ff1de-e31d-4139-b3c5-b73c055d82ca_characterizeImage_845913_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 66 try 0 failed parsl.dataflow.dflow INFO: Task 66 marked for retry parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845975/da1c7d8f-e6f9-46a6-b3f2-8258eaeb2b51_characterizeImage_845975_2.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845975/da1c7d8f-e6f9-46a6-b3f2-8258eaeb2b51_characterizeImage_845975_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 66 has memoization hash 95b9654e07da090311c0189aef5b5d18 parsl.dataflow.memoization INFO: Task 66 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9968860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 66 try 1 launched on executor multi with executor id 137 parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845975/da1c7d8f-e6f9-46a6-b3f2-8258eaeb2b51_characterizeImage_845975_2.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845975/da1c7d8f-e6f9-46a6-b3f2-8258eaeb2b51_characterizeImage_845975_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 64 try 0 failed parsl.dataflow.dflow INFO: Task 64 marked for retry parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845874/4badbe55-5bf8-4c60-9092-cb226d85eda1_characterizeImage_845874_2.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845874/4badbe55-5bf8-4c60-9092-cb226d85eda1_characterizeImage_845874_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash 292107de6482feee20e278a818f6538c parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9968a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 1 launched on executor multi with executor id 138 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845874/4badbe55-5bf8-4c60-9092-cb226d85eda1_characterizeImage_845874_2.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845874/4badbe55-5bf8-4c60-9092-cb226d85eda1_characterizeImage_845874_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 86 try 0 failed parsl.dataflow.dflow INFO: Task 86 marked for retry parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845936/0ad24b81-78f0-4a71-85b1-5897243e3959_characterizeImage_845936_2.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845936/0ad24b81-78f0-4a71-85b1-5897243e3959_characterizeImage_845936_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 86 has memoization hash bea4af1b9022478ce8f2cd1c66f0db0c parsl.dataflow.memoization INFO: Task 86 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9969b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 86 try 1 launched on executor multi with executor id 139 parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845936/0ad24b81-78f0-4a71-85b1-5897243e3959_characterizeImage_845936_2.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845936/0ad24b81-78f0-4a71-85b1-5897243e3959_characterizeImage_845936_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 82 try 0 failed parsl.dataflow.dflow INFO: Task 82 marked for retry parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845883/ea7a2c77-8df1-498a-aa9e-940279db3652_characterizeImage_845883_2.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845883/ea7a2c77-8df1-498a-aa9e-940279db3652_characterizeImage_845883_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash 571deec6cb8c2b574c780b01457b7ed0 parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9969940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 1 launched on executor multi with executor id 140 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845883/ea7a2c77-8df1-498a-aa9e-940279db3652_characterizeImage_845883_2.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845883/ea7a2c77-8df1-498a-aa9e-940279db3652_characterizeImage_845883_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 92 try 0 failed parsl.dataflow.dflow INFO: Task 92 marked for retry parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845907/cffcd057-7872-437d-9715-76380aa988d4_characterizeImage_845907_2.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845907/cffcd057-7872-437d-9715-76380aa988d4_characterizeImage_845907_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash 1a700a3cc267ef0bf4016ba03e69a613 parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de996a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 1 launched on executor multi with executor id 141 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845907/cffcd057-7872-437d-9715-76380aa988d4_characterizeImage_845907_2.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845907/cffcd057-7872-437d-9715-76380aa988d4_characterizeImage_845907_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 70 try 0 failed parsl.dataflow.dflow INFO: Task 70 marked for retry parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845967/5a555250-817e-4342-92e0-7aaa9746b09c_characterizeImage_845967_2.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845967/5a555250-817e-4342-92e0-7aaa9746b09c_characterizeImage_845967_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash 59111f98da0a7bd3e808682dc15c3e52 parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9968e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 1 launched on executor multi with executor id 142 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845967/5a555250-817e-4342-92e0-7aaa9746b09c_characterizeImage_845967_2.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845967/5a555250-817e-4342-92e0-7aaa9746b09c_characterizeImage_845967_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 68 try 0 failed parsl.dataflow.dflow INFO: Task 68 marked for retry parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845945/3a266ace-a6aa-428e-88a7-b3fccd7d2042_characterizeImage_845945_2.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845945/3a266ace-a6aa-428e-88a7-b3fccd7d2042_characterizeImage_845945_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 68 has memoization hash d8ec5db9d7aa4d44b438da3d92317e06 parsl.dataflow.memoization INFO: Task 68 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9968c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 68 try 1 launched on executor multi with executor id 143 parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845945/3a266ace-a6aa-428e-88a7-b3fccd7d2042_characterizeImage_845945_2.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845945/3a266ace-a6aa-428e-88a7-b3fccd7d2042_characterizeImage_845945_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 76 try 0 failed parsl.dataflow.dflow INFO: Task 76 marked for retry parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845885/af1fe9c9-54fc-473f-887d-3a7ae72aa8a3_characterizeImage_845885_2.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845885/af1fe9c9-54fc-473f-887d-3a7ae72aa8a3_characterizeImage_845885_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash 6731d816e8d8239d3dbf0561c8423f22 parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99691c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 1 launched on executor multi with executor id 144 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845885/af1fe9c9-54fc-473f-887d-3a7ae72aa8a3_characterizeImage_845885_2.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845885/af1fe9c9-54fc-473f-887d-3a7ae72aa8a3_characterizeImage_845885_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 80 try 0 failed parsl.dataflow.dflow INFO: Task 80 marked for retry parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845957/7aa5b53f-f329-414b-9b47-d7b6f9e13666_characterizeImage_845957_2.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845957/7aa5b53f-f329-414b-9b47-d7b6f9e13666_characterizeImage_845957_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 80 has memoization hash b09277256f23afd0bad289210c29570a parsl.dataflow.memoization INFO: Task 80 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9969760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 80 try 1 launched on executor multi with executor id 145 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845957/7aa5b53f-f329-414b-9b47-d7b6f9e13666_characterizeImage_845957_2.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845957/7aa5b53f-f329-414b-9b47-d7b6f9e13666_characterizeImage_845957_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 74 try 0 failed parsl.dataflow.dflow INFO: Task 74 marked for retry parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845964/303de284-995e-41c2-8c35-f444be467aaa_characterizeImage_845964_2.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845964/303de284-995e-41c2-8c35-f444be467aaa_characterizeImage_845964_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 74 has memoization hash 54617f18a55d67a4158e9f7f5c124cce parsl.dataflow.memoization INFO: Task 74 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99693a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 74 try 1 launched on executor multi with executor id 146 parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845964/303de284-995e-41c2-8c35-f444be467aaa_characterizeImage_845964_2.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845964/303de284-995e-41c2-8c35-f444be467aaa_characterizeImage_845964_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 88 try 0 failed parsl.dataflow.dflow INFO: Task 88 marked for retry parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845947/f101aedf-9483-455c-b975-a91bbd2761c6_characterizeImage_845947_2.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845947/f101aedf-9483-455c-b975-a91bbd2761c6_characterizeImage_845947_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 88 has memoization hash 41840f84335b33e2575117574502b150 parsl.dataflow.memoization INFO: Task 88 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9969ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 88 try 1 launched on executor multi with executor id 147 parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845947/f101aedf-9483-455c-b975-a91bbd2761c6_characterizeImage_845947_2.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845947/f101aedf-9483-455c-b975-a91bbd2761c6_characterizeImage_845947_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 94 try 0 failed parsl.dataflow.dflow INFO: Task 94 marked for retry parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845899/56e7a245-664c-40c0-bd2b-24779e8ad2e0_characterizeImage_845899_2.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845899/56e7a245-664c-40c0-bd2b-24779e8ad2e0_characterizeImage_845899_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 94 has memoization hash 1152693677a0369f82a87ee6d585cc04 parsl.dataflow.memoization INFO: Task 94 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de996a660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 94 try 1 launched on executor multi with executor id 148 parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845899/56e7a245-664c-40c0-bd2b-24779e8ad2e0_characterizeImage_845899_2.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845899/56e7a245-664c-40c0-bd2b-24779e8ad2e0_characterizeImage_845899_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 72 try 0 failed parsl.dataflow.dflow INFO: Task 72 marked for retry parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845918/0b009a5e-e046-4261-8fa5-3c38569211ba_characterizeImage_845918_2.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845918/0b009a5e-e046-4261-8fa5-3c38569211ba_characterizeImage_845918_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 72 has memoization hash 78202a5dd2e162448f52d6908381c4c3 parsl.dataflow.memoization INFO: Task 72 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9968fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 72 try 1 launched on executor multi with executor id 149 parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845918/0b009a5e-e046-4261-8fa5-3c38569211ba_characterizeImage_845918_2.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845918/0b009a5e-e046-4261-8fa5-3c38569211ba_characterizeImage_845918_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 78 try 0 failed parsl.dataflow.dflow INFO: Task 78 marked for retry parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845941/0c7af297-2442-4256-b0b6-1c9c478a6c29_characterizeImage_845941_2.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845941/0c7af297-2442-4256-b0b6-1c9c478a6c29_characterizeImage_845941_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 78 has memoization hash fe49a52810bd6bb64f918aabbddc662a parsl.dataflow.memoization INFO: Task 78 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9969580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 78 try 1 launched on executor multi with executor id 150 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845941/0c7af297-2442-4256-b0b6-1c9c478a6c29_characterizeImage_845941_2.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845941/0c7af297-2442-4256-b0b6-1c9c478a6c29_characterizeImage_845941_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 84 try 0 failed parsl.dataflow.dflow INFO: Task 84 marked for retry parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845919/f5d0cb70-b110-4afa-b2a2-4a6f4f1450cf_characterizeImage_845919_2.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845919/f5d0cb70-b110-4afa-b2a2-4a6f4f1450cf_characterizeImage_845919_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 84 has memoization hash 5edcb9752b5fda47225a99f5e0a8d9c8 parsl.dataflow.memoization INFO: Task 84 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9969d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 84 try 1 launched on executor multi with executor id 151 parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845919/f5d0cb70-b110-4afa-b2a2-4a6f4f1450cf_characterizeImage_845919_2.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845919/f5d0cb70-b110-4afa-b2a2-4a6f4f1450cf_characterizeImage_845919_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 90 try 0 failed parsl.dataflow.dflow INFO: Task 90 marked for retry parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845896/b046a7ec-89a9-45b3-ab8f-876092c79520_characterizeImage_845896_2.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845896/b046a7ec-89a9-45b3-ab8f-876092c79520_characterizeImage_845896_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 90 has memoization hash 35cad0d2e8de02aa550181216d5111dc parsl.dataflow.memoization INFO: Task 90 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de996a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 90 try 1 launched on executor multi with executor id 152 parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845896/b046a7ec-89a9-45b3-ab8f-876092c79520_characterizeImage_845896_2.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845896/b046a7ec-89a9-45b3-ab8f-876092c79520_characterizeImage_845896_2.stderr 2024-03-13 00:15:42 proc_lsst.multi:146 [INFO] found job 10228 in provider local proc_lsst.multi INFO: found job 10228 in provider local 2024-03-13 00:15:42 proc_lsst.multi:146 [INFO] found job 17006354 in provider astro proc_lsst.multi INFO: found job 17006354 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:15:42 proc_lsst.multi:146 [INFO] found job 17006355 in provider astro proc_lsst.multi INFO: found job 17006355 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:15:43 proc_lsst.multi:146 [INFO] found job 17006356 in provider astro proc_lsst.multi INFO: found job 17006356 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:15:43 proc_lsst.multi:146 [INFO] found job 17006357 in provider astro proc_lsst.multi INFO: found job 17006357 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 96 try 0 failed parsl.dataflow.dflow INFO: Task 96 marked for retry parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845908/2cccf77c-c5bd-400f-b9f6-e902f3419685_characterizeImage_845908_2.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845908/2cccf77c-c5bd-400f-b9f6-e902f3419685_characterizeImage_845908_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 96 has memoization hash d970371a5a4b416dc608b556d1ed8ed2 parsl.dataflow.memoization INFO: Task 96 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de996a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 96 try 1 launched on executor multi with executor id 153 parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845908/2cccf77c-c5bd-400f-b9f6-e902f3419685_characterizeImage_845908_2.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845908/2cccf77c-c5bd-400f-b9f6-e902f3419685_characterizeImage_845908_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 98 try 0 failed parsl.dataflow.dflow INFO: Task 98 marked for retry parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845958/d60f3d61-9a95-457d-97fe-f040390eb6f8_characterizeImage_845958_2.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845958/d60f3d61-9a95-457d-97fe-f040390eb6f8_characterizeImage_845958_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 98 has memoization hash bba3b5a3efc393e35738df156d37271f parsl.dataflow.memoization INFO: Task 98 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de996a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 98 try 1 launched on executor multi with executor id 154 parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845958/d60f3d61-9a95-457d-97fe-f040390eb6f8_characterizeImage_845958_2.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845958/d60f3d61-9a95-457d-97fe-f040390eb6f8_characterizeImage_845958_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:15:44 proc_lsst.multi:146 [INFO] found job 17006358 in provider astro proc_lsst.multi INFO: found job 17006358 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:15:44 proc_lsst.multi:146 [INFO] found job 17006359 in provider astro proc_lsst.multi INFO: found job 17006359 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:15:45 proc_lsst.multi:146 [INFO] found job 17006360 in provider astro proc_lsst.multi INFO: found job 17006360 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:15:45 proc_lsst.multi:146 [INFO] found job 17006361 in provider astro proc_lsst.multi INFO: found job 17006361 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:15:46 proc_lsst.multi:146 [INFO] found job 17006362 in provider astro proc_lsst.multi INFO: found job 17006362 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 102 try 0 failed parsl.dataflow.dflow INFO: Task 102 marked for retry parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845961/7727537b-7d07-45a6-9122-acbcb9e077ce_characterizeImage_845961_2.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845961/7727537b-7d07-45a6-9122-acbcb9e077ce_characterizeImage_845961_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 102 has memoization hash b17a8e6300c09938af380732f8a8f6e4 parsl.dataflow.memoization INFO: Task 102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de996ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 102 try 1 launched on executor multi with executor id 155 parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845961/7727537b-7d07-45a6-9122-acbcb9e077ce_characterizeImage_845961_2.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845961/7727537b-7d07-45a6-9122-acbcb9e077ce_characterizeImage_845961_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 100 try 0 failed parsl.dataflow.dflow INFO: Task 100 marked for retry parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845880/abd6192e-1c66-4408-9e4c-dec63d0b40ab_characterizeImage_845880_2.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845880/abd6192e-1c66-4408-9e4c-dec63d0b40ab_characterizeImage_845880_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash 2a005eae06a1644529e6f392ef4ef4c1 parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de996aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 1 launched on executor multi with executor id 156 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845880/abd6192e-1c66-4408-9e4c-dec63d0b40ab_characterizeImage_845880_2.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845880/abd6192e-1c66-4408-9e4c-dec63d0b40ab_characterizeImage_845880_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 104 try 0 failed parsl.dataflow.dflow INFO: Task 104 marked for retry parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845946/1a01a87b-7ea3-4ff9-97de-18786cddcb4e_characterizeImage_845946_2.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845946/1a01a87b-7ea3-4ff9-97de-18786cddcb4e_characterizeImage_845946_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 104 has memoization hash 26a197c55a718db18c438ed6ab2c43d0 parsl.dataflow.memoization INFO: Task 104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de996afc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 104 try 1 launched on executor multi with executor id 157 parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845946/1a01a87b-7ea3-4ff9-97de-18786cddcb4e_characterizeImage_845946_2.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845946/1a01a87b-7ea3-4ff9-97de-18786cddcb4e_characterizeImage_845946_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 106 try 0 failed parsl.dataflow.dflow INFO: Task 106 marked for retry parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845897/db4180e0-e7ad-4ce2-a1f2-4ce12f40d7d5_characterizeImage_845897_2.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845897/db4180e0-e7ad-4ce2-a1f2-4ce12f40d7d5_characterizeImage_845897_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash 56f9b34c1d6d2fae87614ce08fb1d76a parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de996ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 106 try 1 launched on executor multi with executor id 158 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845897/db4180e0-e7ad-4ce2-a1f2-4ce12f40d7d5_characterizeImage_845897_2.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845897/db4180e0-e7ad-4ce2-a1f2-4ce12f40d7d5_characterizeImage_845897_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:15:46 proc_lsst.multi:146 [INFO] found job 17006363 in provider astro proc_lsst.multi INFO: found job 17006363 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:15:47 proc_lsst.multi:146 [INFO] found job 17006364 in provider astro proc_lsst.multi INFO: found job 17006364 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:15:47 proc_lsst.multi:146 [INFO] found job 17006365 in provider astro proc_lsst.multi INFO: found job 17006365 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:15:48 proc_lsst.multi:146 [INFO] found job 17006366 in provider astro proc_lsst.multi INFO: found job 17006366 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 126 try 0 failed parsl.dataflow.dflow INFO: Task 126 marked for retry parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845894/310d287a-0551-48bb-8f2d-6afe4b4d8367_characterizeImage_845894_2.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845894/310d287a-0551-48bb-8f2d-6afe4b4d8367_characterizeImage_845894_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 126 has memoization hash 634b7baf979a5f735559d77d3a9ec9ca parsl.dataflow.memoization INFO: Task 126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99940e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 126 try 1 launched on executor multi with executor id 159 parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845894/310d287a-0551-48bb-8f2d-6afe4b4d8367_characterizeImage_845894_2.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845894/310d287a-0551-48bb-8f2d-6afe4b4d8367_characterizeImage_845894_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 114 try 0 failed parsl.dataflow.dflow INFO: Task 114 marked for retry parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845965/c6c0e87d-e3fc-4858-bef7-c19e9f1aea39_characterizeImage_845965_2.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845965/c6c0e87d-e3fc-4858-bef7-c19e9f1aea39_characterizeImage_845965_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 114 has memoization hash 05e5a6aaca09e363ebff001ce54648f8 parsl.dataflow.memoization INFO: Task 114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de996b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 114 try 1 launched on executor multi with executor id 160 parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845965/c6c0e87d-e3fc-4858-bef7-c19e9f1aea39_characterizeImage_845965_2.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845965/c6c0e87d-e3fc-4858-bef7-c19e9f1aea39_characterizeImage_845965_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 124 try 0 failed parsl.dataflow.dflow INFO: Task 124 marked for retry parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845943/3d4b9af2-e295-4738-98ac-d7a871c68578_characterizeImage_845943_2.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845943/3d4b9af2-e295-4738-98ac-d7a871c68578_characterizeImage_845943_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 124 has memoization hash 28c8cdbabcc323c8e97ebf56b860f027 parsl.dataflow.memoization INFO: Task 124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99942c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 124 try 1 launched on executor multi with executor id 161 parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845943/3d4b9af2-e295-4738-98ac-d7a871c68578_characterizeImage_845943_2.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845943/3d4b9af2-e295-4738-98ac-d7a871c68578_characterizeImage_845943_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 128 try 0 failed parsl.dataflow.dflow INFO: Task 128 marked for retry parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845902/33330bca-6bd1-4857-85da-515b4f803089_characterizeImage_845902_2.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845902/33330bca-6bd1-4857-85da-515b4f803089_characterizeImage_845902_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 128 has memoization hash da349ef6b94b0ceaf780a692c10ee37a parsl.dataflow.memoization INFO: Task 128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99944a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 128 try 1 launched on executor multi with executor id 162 parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845902/33330bca-6bd1-4857-85da-515b4f803089_characterizeImage_845902_2.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845902/33330bca-6bd1-4857-85da-515b4f803089_characterizeImage_845902_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 138 try 0 failed parsl.dataflow.dflow INFO: Task 138 marked for retry parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845942/654704d2-6c3f-4dd2-a260-0a17b3b3d2ad_characterizeImage_845942_2.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845942/654704d2-6c3f-4dd2-a260-0a17b3b3d2ad_characterizeImage_845942_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 138 has memoization hash a464cc6d36b1e40e30e451fcabdd05fb parsl.dataflow.memoization INFO: Task 138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9994e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 138 try 1 launched on executor multi with executor id 163 parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845942/654704d2-6c3f-4dd2-a260-0a17b3b3d2ad_characterizeImage_845942_2.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845942/654704d2-6c3f-4dd2-a260-0a17b3b3d2ad_characterizeImage_845942_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 134 try 0 failed parsl.dataflow.dflow INFO: Task 134 marked for retry parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845933/e24cc46c-28c5-4500-9354-c09907f1ab1a_characterizeImage_845933_2.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845933/e24cc46c-28c5-4500-9354-c09907f1ab1a_characterizeImage_845933_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 134 has memoization hash 4cb6affb9a82af1e1e6752ca4a88faf0 parsl.dataflow.memoization INFO: Task 134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9994c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 134 try 1 launched on executor multi with executor id 164 parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845933/e24cc46c-28c5-4500-9354-c09907f1ab1a_characterizeImage_845933_2.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845933/e24cc46c-28c5-4500-9354-c09907f1ab1a_characterizeImage_845933_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 118 try 0 failed parsl.dataflow.dflow INFO: Task 118 marked for retry parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845921/5199ae07-c639-49d5-8195-de5cfcf37276_characterizeImage_845921_2.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845921/5199ae07-c639-49d5-8195-de5cfcf37276_characterizeImage_845921_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash f1502731263e78151067c062569940af parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de996bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 1 launched on executor multi with executor id 165 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845921/5199ae07-c639-49d5-8195-de5cfcf37276_characterizeImage_845921_2.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845921/5199ae07-c639-49d5-8195-de5cfcf37276_characterizeImage_845921_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 122 try 0 failed parsl.dataflow.dflow INFO: Task 122 marked for retry parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845931/24c0bb7d-7496-4719-a33c-c8419935b473_characterizeImage_845931_2.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845931/24c0bb7d-7496-4719-a33c-c8419935b473_characterizeImage_845931_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 122 has memoization hash 8be815d32965502f9ba5eaf4f1f0e28c parsl.dataflow.memoization INFO: Task 122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9994040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 122 try 1 launched on executor multi with executor id 166 parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845931/24c0bb7d-7496-4719-a33c-c8419935b473_characterizeImage_845931_2.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845931/24c0bb7d-7496-4719-a33c-c8419935b473_characterizeImage_845931_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 120 try 0 failed parsl.dataflow.dflow INFO: Task 120 marked for retry parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845944/a6044269-9d83-4d2b-b2c5-0d2f8fd837c1_characterizeImage_845944_2.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845944/a6044269-9d83-4d2b-b2c5-0d2f8fd837c1_characterizeImage_845944_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 120 has memoization hash 39fa3e7c682bb37e7b54f33bc6e4fc55 parsl.dataflow.memoization INFO: Task 120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de996bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 120 try 1 launched on executor multi with executor id 167 parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845944/a6044269-9d83-4d2b-b2c5-0d2f8fd837c1_characterizeImage_845944_2.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845944/a6044269-9d83-4d2b-b2c5-0d2f8fd837c1_characterizeImage_845944_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 136 try 0 failed parsl.dataflow.dflow INFO: Task 136 marked for retry parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845873/c4125fac-ad41-4766-890a-5eb55c5d1ff6_characterizeImage_845873_2.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845873/c4125fac-ad41-4766-890a-5eb55c5d1ff6_characterizeImage_845873_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 136 has memoization hash fa8677c0e62d828621eed9abfac2961e parsl.dataflow.memoization INFO: Task 136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9994a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 136 try 1 launched on executor multi with executor id 168 parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845873/c4125fac-ad41-4766-890a-5eb55c5d1ff6_characterizeImage_845873_2.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845873/c4125fac-ad41-4766-890a-5eb55c5d1ff6_characterizeImage_845873_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 112 try 0 failed parsl.dataflow.dflow INFO: Task 112 marked for retry parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845901/238275a5-1580-4fe5-874f-03519b7a35f4_characterizeImage_845901_2.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845901/238275a5-1580-4fe5-874f-03519b7a35f4_characterizeImage_845901_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 112 has memoization hash 2937485724674bc21461a4ffe8d183ff parsl.dataflow.memoization INFO: Task 112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de996b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 112 try 1 launched on executor multi with executor id 169 parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845901/238275a5-1580-4fe5-874f-03519b7a35f4_characterizeImage_845901_2.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845901/238275a5-1580-4fe5-874f-03519b7a35f4_characterizeImage_845901_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 130 try 0 failed parsl.dataflow.dflow INFO: Task 130 marked for retry parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845915/6203f0d5-9054-4686-8b95-faef0a5edb06_characterizeImage_845915_2.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845915/6203f0d5-9054-4686-8b95-faef0a5edb06_characterizeImage_845915_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 130 has memoization hash d7977db724d22a2451ef2d979c1a6c27 parsl.dataflow.memoization INFO: Task 130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9994680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 130 try 1 launched on executor multi with executor id 170 parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845915/6203f0d5-9054-4686-8b95-faef0a5edb06_characterizeImage_845915_2.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845915/6203f0d5-9054-4686-8b95-faef0a5edb06_characterizeImage_845915_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 116 try 0 failed parsl.dataflow.dflow INFO: Task 116 marked for retry parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845960/0c21b0b0-2e4e-4feb-9471-2c58334fc3d3_characterizeImage_845960_2.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845960/0c21b0b0-2e4e-4feb-9471-2c58334fc3d3_characterizeImage_845960_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 116 has memoization hash 58a5f1323cffab8aba0b198436379bbf parsl.dataflow.memoization INFO: Task 116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de996b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 116 try 1 launched on executor multi with executor id 171 parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845960/0c21b0b0-2e4e-4feb-9471-2c58334fc3d3_characterizeImage_845960_2.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845960/0c21b0b0-2e4e-4feb-9471-2c58334fc3d3_characterizeImage_845960_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 132 try 0 failed parsl.dataflow.dflow INFO: Task 132 marked for retry parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845910/b6a1b982-bd05-4e4f-99e1-3adbfca0c6aa_characterizeImage_845910_2.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845910/b6a1b982-bd05-4e4f-99e1-3adbfca0c6aa_characterizeImage_845910_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 132 has memoization hash 5f1f9fea53aa5aeb4a8a20beb0ce6740 parsl.dataflow.memoization INFO: Task 132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9994860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 132 try 1 launched on executor multi with executor id 172 parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845910/b6a1b982-bd05-4e4f-99e1-3adbfca0c6aa_characterizeImage_845910_2.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845910/b6a1b982-bd05-4e4f-99e1-3adbfca0c6aa_characterizeImage_845910_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:15:48 proc_lsst.multi:146 [INFO] found job 17006367 in provider astro proc_lsst.multi INFO: found job 17006367 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 110 try 0 failed parsl.dataflow.dflow INFO: Task 110 marked for retry parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845934/6d57084e-9f7b-4001-a687-2e72889d80fe_characterizeImage_845934_2.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845934/6d57084e-9f7b-4001-a687-2e72889d80fe_characterizeImage_845934_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 110 has memoization hash 1b4d13183092be934a16bef0db0574f0 parsl.dataflow.memoization INFO: Task 110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de996b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 110 try 1 launched on executor multi with executor id 173 parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845934/6d57084e-9f7b-4001-a687-2e72889d80fe_characterizeImage_845934_2.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845934/6d57084e-9f7b-4001-a687-2e72889d80fe_characterizeImage_845934_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:15:49 proc_lsst.multi:146 [INFO] found job 17006368 in provider astro proc_lsst.multi INFO: found job 17006368 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 108 try 0 failed parsl.dataflow.dflow INFO: Task 108 marked for retry parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845890/2b57e739-0b5a-4f52-a6dd-18fe97769482_characterizeImage_845890_2.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845890/2b57e739-0b5a-4f52-a6dd-18fe97769482_characterizeImage_845890_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 108 has memoization hash b21401054395f5f8ba48adc9410115f2 parsl.dataflow.memoization INFO: Task 108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de996b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 108 try 1 launched on executor multi with executor id 174 parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845890/2b57e739-0b5a-4f52-a6dd-18fe97769482_characterizeImage_845890_2.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845890/2b57e739-0b5a-4f52-a6dd-18fe97769482_characterizeImage_845890_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:15:49 proc_lsst.multi:146 [INFO] found job 17006369 in provider astro proc_lsst.multi INFO: found job 17006369 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:15:50 proc_lsst.multi:146 [INFO] found job 17006370 in provider astro proc_lsst.multi INFO: found job 17006370 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:15:50 proc_lsst.multi:146 [INFO] found job 17006371 in provider astro proc_lsst.multi INFO: found job 17006371 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:15:51 proc_lsst.multi:146 [INFO] found job 17006372 in provider astro proc_lsst.multi INFO: found job 17006372 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:15:51 proc_lsst.multi:146 [INFO] found job 17006373 in provider astro proc_lsst.multi INFO: found job 17006373 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 140 try 0 failed parsl.dataflow.dflow INFO: Task 140 marked for retry parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845970/b5026a35-dc05-4010-b148-df5c3b5770b0_characterizeImage_845970_2.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845970/b5026a35-dc05-4010-b148-df5c3b5770b0_characterizeImage_845970_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 140 has memoization hash 2efcf3d6aaab8c7d58eab75596239da6 parsl.dataflow.memoization INFO: Task 140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9994fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 140 try 1 launched on executor multi with executor id 175 parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845970/b5026a35-dc05-4010-b148-df5c3b5770b0_characterizeImage_845970_2.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845970/b5026a35-dc05-4010-b148-df5c3b5770b0_characterizeImage_845970_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:15:52 proc_lsst.multi:146 [INFO] found job 17006374 in provider astro proc_lsst.multi INFO: found job 17006374 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:15:52 proc_lsst.multi:146 [INFO] found job 17006375 in provider astro proc_lsst.multi INFO: found job 17006375 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 142 try 0 failed parsl.dataflow.dflow INFO: Task 142 marked for retry parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845971/85c8671d-b56e-4084-b9a7-e0c7fc16a71d_characterizeImage_845971_2.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845971/85c8671d-b56e-4084-b9a7-e0c7fc16a71d_characterizeImage_845971_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 142 has memoization hash 00b56b1de7658aa4bc1562e0d4c859fc parsl.dataflow.memoization INFO: Task 142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99951c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 142 try 1 launched on executor multi with executor id 176 parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845971/85c8671d-b56e-4084-b9a7-e0c7fc16a71d_characterizeImage_845971_2.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845971/85c8671d-b56e-4084-b9a7-e0c7fc16a71d_characterizeImage_845971_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:15:53 proc_lsst.multi:146 [INFO] found job 17006376 in provider astro proc_lsst.multi INFO: found job 17006376 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 144 try 0 failed parsl.dataflow.dflow INFO: Task 144 marked for retry parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845940/8242da42-4187-4773-a9dd-eaa55e453baa_characterizeImage_845940_2.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845940/8242da42-4187-4773-a9dd-eaa55e453baa_characterizeImage_845940_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 144 has memoization hash af259f7fcd1ef5dfbc0433c91cc0a582 parsl.dataflow.memoization INFO: Task 144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9995580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 144 try 1 launched on executor multi with executor id 177 parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845940/8242da42-4187-4773-a9dd-eaa55e453baa_characterizeImage_845940_2.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845940/8242da42-4187-4773-a9dd-eaa55e453baa_characterizeImage_845940_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 146 try 0 failed parsl.dataflow.dflow INFO: Task 146 marked for retry parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845887/1b07f960-0e34-4d29-bf40-2efafa1ec813_characterizeImage_845887_2.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845887/1b07f960-0e34-4d29-bf40-2efafa1ec813_characterizeImage_845887_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 146 has memoization hash 8e4689bdd3a43f6074fc3be7d05ad7c0 parsl.dataflow.memoization INFO: Task 146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99953a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 146 try 1 launched on executor multi with executor id 178 parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845887/1b07f960-0e34-4d29-bf40-2efafa1ec813_characterizeImage_845887_2.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845887/1b07f960-0e34-4d29-bf40-2efafa1ec813_characterizeImage_845887_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:15:53 proc_lsst.multi:146 [INFO] found job 17006377 in provider astro proc_lsst.multi INFO: found job 17006377 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 148 try 0 failed parsl.dataflow.dflow INFO: Task 148 marked for retry parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845937/ee634d28-bb24-424f-8807-5913155a88bb_characterizeImage_845937_2.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845937/ee634d28-bb24-424f-8807-5913155a88bb_characterizeImage_845937_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 148 has memoization hash e291a612bb8e610169bc870795eca5dc parsl.dataflow.memoization INFO: Task 148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9995760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 148 try 1 launched on executor multi with executor id 179 parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845937/ee634d28-bb24-424f-8807-5913155a88bb_characterizeImage_845937_2.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845937/ee634d28-bb24-424f-8807-5913155a88bb_characterizeImage_845937_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 150 try 0 failed parsl.dataflow.dflow INFO: Task 150 marked for retry parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845878/d5f15356-f090-4ae5-8e64-7f0360398e16_characterizeImage_845878_2.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845878/d5f15356-f090-4ae5-8e64-7f0360398e16_characterizeImage_845878_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 150 has memoization hash 9d9f8f84922b633fc8f11ababe429bd0 parsl.dataflow.memoization INFO: Task 150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9995940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 150 try 1 launched on executor multi with executor id 180 parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845878/d5f15356-f090-4ae5-8e64-7f0360398e16_characterizeImage_845878_2.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845878/d5f15356-f090-4ae5-8e64-7f0360398e16_characterizeImage_845878_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:15:54 proc_lsst.multi:146 [INFO] found job 17006378 in provider astro proc_lsst.multi INFO: found job 17006378 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 26, active_tasks = 104 parsl.dataflow.strategy DEBUG: Executor multi has 104 active tasks, 26/0 running/pending blocks, and 27 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23218217990800 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 26, active_tasks = 104 parsl.dataflow.strategy DEBUG: Executor multi has 104 active tasks, 26/0 running/pending blocks, and 27 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23218217990800 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 164 try 0 failed parsl.dataflow.dflow INFO: Task 164 marked for retry parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845898/28352836-5c66-4e8b-a3bb-a7128f2e6b3c_characterizeImage_845898_2.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845898/28352836-5c66-4e8b-a3bb-a7128f2e6b3c_characterizeImage_845898_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 164 has memoization hash cbf9ce955b61128a561db8b181be4d0f parsl.dataflow.memoization INFO: Task 164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9996840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 164 try 1 launched on executor multi with executor id 181 parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845898/28352836-5c66-4e8b-a3bb-a7128f2e6b3c_characterizeImage_845898_2.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845898/28352836-5c66-4e8b-a3bb-a7128f2e6b3c_characterizeImage_845898_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 162 try 0 failed parsl.dataflow.dflow INFO: Task 162 marked for retry parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845903/da3e7f1f-c1f6-4f7b-a2f3-02a28506d989_characterizeImage_845903_2.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845903/da3e7f1f-c1f6-4f7b-a2f3-02a28506d989_characterizeImage_845903_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 162 has memoization hash 2f450815e21e44b313d3883b69ad19e3 parsl.dataflow.memoization INFO: Task 162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9996480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 162 try 1 launched on executor multi with executor id 182 parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845903/da3e7f1f-c1f6-4f7b-a2f3-02a28506d989_characterizeImage_845903_2.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845903/da3e7f1f-c1f6-4f7b-a2f3-02a28506d989_characterizeImage_845903_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 182 try 0 failed parsl.dataflow.dflow INFO: Task 182 marked for retry parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845950/6f1a7668-a044-4ff6-b762-1a6e2bfc97f2_characterizeImage_845950_2.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845950/6f1a7668-a044-4ff6-b762-1a6e2bfc97f2_characterizeImage_845950_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 182 has memoization hash 83c32e90a036e41a982dc399a7d6f5af parsl.dataflow.memoization INFO: Task 182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9997740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 182 try 1 launched on executor multi with executor id 183 parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845950/6f1a7668-a044-4ff6-b762-1a6e2bfc97f2_characterizeImage_845950_2.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845950/6f1a7668-a044-4ff6-b762-1a6e2bfc97f2_characterizeImage_845950_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 168 try 0 failed parsl.dataflow.dflow INFO: Task 168 marked for retry parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845875/9bc5aca3-17c2-45c9-9565-5995c7db1735_characterizeImage_845875_2.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845875/9bc5aca3-17c2-45c9-9565-5995c7db1735_characterizeImage_845875_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 168 has memoization hash 84f94359e378d4bd0ce4dbcf2dd93e4a parsl.dataflow.memoization INFO: Task 168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9996a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 168 try 1 launched on executor multi with executor id 184 parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845875/9bc5aca3-17c2-45c9-9565-5995c7db1735_characterizeImage_845875_2.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845875/9bc5aca3-17c2-45c9-9565-5995c7db1735_characterizeImage_845875_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 178 try 0 failed parsl.dataflow.dflow INFO: Task 178 marked for retry parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845900/488d80a1-d095-4def-b45a-12fc69abfa32_characterizeImage_845900_2.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845900/488d80a1-d095-4def-b45a-12fc69abfa32_characterizeImage_845900_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 178 has memoization hash 28643aa54d498a59227ecef6d1e72523 parsl.dataflow.memoization INFO: Task 178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9997380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 178 try 1 launched on executor multi with executor id 185 parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845900/488d80a1-d095-4def-b45a-12fc69abfa32_characterizeImage_845900_2.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845900/488d80a1-d095-4def-b45a-12fc69abfa32_characterizeImage_845900_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 176 try 0 failed parsl.dataflow.dflow INFO: Task 176 marked for retry parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845872/f877b180-ff8a-4843-b8b3-6f68ec528560_characterizeImage_845872_2.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845872/f877b180-ff8a-4843-b8b3-6f68ec528560_characterizeImage_845872_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 176 has memoization hash 5f1bee9a3d168108278ed93378da91d3 parsl.dataflow.memoization INFO: Task 176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9996fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 176 try 1 launched on executor multi with executor id 186 parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845872/f877b180-ff8a-4843-b8b3-6f68ec528560_characterizeImage_845872_2.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845872/f877b180-ff8a-4843-b8b3-6f68ec528560_characterizeImage_845872_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 166 try 0 failed parsl.dataflow.dflow INFO: Task 166 marked for retry parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845884/cfcb4712-11f0-43a3-90d5-569ed4c00657_characterizeImage_845884_2.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845884/cfcb4712-11f0-43a3-90d5-569ed4c00657_characterizeImage_845884_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 166 has memoization hash 6018698b71a085912efb29d8c31fb68b parsl.dataflow.memoization INFO: Task 166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9996660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 166 try 1 launched on executor multi with executor id 187 parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845884/cfcb4712-11f0-43a3-90d5-569ed4c00657_characterizeImage_845884_2.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845884/cfcb4712-11f0-43a3-90d5-569ed4c00657_characterizeImage_845884_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 170 try 0 failed parsl.dataflow.dflow INFO: Task 170 marked for retry parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845904/5f371e5b-014c-435d-bf1e-95319d9c08a8_characterizeImage_845904_2.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845904/5f371e5b-014c-435d-bf1e-95319d9c08a8_characterizeImage_845904_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 170 has memoization hash cf501f8bed69538cab5578a17074a0ee parsl.dataflow.memoization INFO: Task 170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9996c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 170 try 1 launched on executor multi with executor id 188 parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845904/5f371e5b-014c-435d-bf1e-95319d9c08a8_characterizeImage_845904_2.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845904/5f371e5b-014c-435d-bf1e-95319d9c08a8_characterizeImage_845904_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 174 try 0 failed parsl.dataflow.dflow INFO: Task 174 marked for retry parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845882/dd838f9a-4c46-46e4-a945-5f2a3d305458_characterizeImage_845882_2.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845882/dd838f9a-4c46-46e4-a945-5f2a3d305458_characterizeImage_845882_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 174 has memoization hash 795e44e2a3bf76947633282bb3375863 parsl.dataflow.memoization INFO: Task 174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99971a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 174 try 1 launched on executor multi with executor id 189 parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845882/dd838f9a-4c46-46e4-a945-5f2a3d305458_characterizeImage_845882_2.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845882/dd838f9a-4c46-46e4-a945-5f2a3d305458_characterizeImage_845882_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 172 try 0 failed parsl.dataflow.dflow INFO: Task 172 marked for retry parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845881/1ef47dee-dced-4ac3-9577-1603c7c7b18a_characterizeImage_845881_2.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845881/1ef47dee-dced-4ac3-9577-1603c7c7b18a_characterizeImage_845881_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 172 has memoization hash dbebc46a58a8c7b92a3991cf1867c43b parsl.dataflow.memoization INFO: Task 172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9996de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 172 try 1 launched on executor multi with executor id 190 parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845881/1ef47dee-dced-4ac3-9577-1603c7c7b18a_characterizeImage_845881_2.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845881/1ef47dee-dced-4ac3-9577-1603c7c7b18a_characterizeImage_845881_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 180 try 0 failed parsl.dataflow.dflow INFO: Task 180 marked for retry parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845922/eb971c87-0a6c-4a04-9d00-74a1f966b764_characterizeImage_845922_2.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845922/eb971c87-0a6c-4a04-9d00-74a1f966b764_characterizeImage_845922_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 180 has memoization hash dfd9556d2e7a174dcdd1fbe33721d23e parsl.dataflow.memoization INFO: Task 180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9997560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 180 try 1 launched on executor multi with executor id 191 parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845922/eb971c87-0a6c-4a04-9d00-74a1f966b764_characterizeImage_845922_2.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845922/eb971c87-0a6c-4a04-9d00-74a1f966b764_characterizeImage_845922_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 184 try 0 failed parsl.dataflow.dflow INFO: Task 184 marked for retry parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845952/2d5dcdc4-3ad9-4a50-9879-e6bf6c638b35_characterizeImage_845952_2.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845952/2d5dcdc4-3ad9-4a50-9879-e6bf6c638b35_characterizeImage_845952_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 184 has memoization hash 6be35a8648e1a0dc2874bb5e842da878 parsl.dataflow.memoization INFO: Task 184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9997b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 184 try 1 launched on executor multi with executor id 192 parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845952/2d5dcdc4-3ad9-4a50-9879-e6bf6c638b35_characterizeImage_845952_2.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845952/2d5dcdc4-3ad9-4a50-9879-e6bf6c638b35_characterizeImage_845952_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 186 try 0 failed parsl.dataflow.dflow INFO: Task 186 marked for retry parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845948/d09b2cb0-abb4-4627-a85a-84f4a7ef645a_characterizeImage_845948_2.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845948/d09b2cb0-abb4-4627-a85a-84f4a7ef645a_characterizeImage_845948_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 186 has memoization hash 785d72514708161c47089ddda2d9876b parsl.dataflow.memoization INFO: Task 186 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9997920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 186 try 1 launched on executor multi with executor id 193 parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845948/d09b2cb0-abb4-4627-a85a-84f4a7ef645a_characterizeImage_845948_2.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845948/d09b2cb0-abb4-4627-a85a-84f4a7ef645a_characterizeImage_845948_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 188 try 0 failed parsl.dataflow.dflow INFO: Task 188 marked for retry parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845959/68a938ce-bc8c-4b0b-9e09-71045e95c554_characterizeImage_845959_2.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845959/68a938ce-bc8c-4b0b-9e09-71045e95c554_characterizeImage_845959_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 188 has memoization hash ab9ec5bc927e597b0f919b042eb5fe90 parsl.dataflow.memoization INFO: Task 188 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9997ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 188 try 1 launched on executor multi with executor id 194 parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845959/68a938ce-bc8c-4b0b-9e09-71045e95c554_characterizeImage_845959_2.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845959/68a938ce-bc8c-4b0b-9e09-71045e95c554_characterizeImage_845959_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 190 try 0 failed parsl.dataflow.dflow INFO: Task 190 marked for retry parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845930/f42fccef-f24c-44d0-83d0-c0b1dc938d0d_characterizeImage_845930_2.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845930/f42fccef-f24c-44d0-83d0-c0b1dc938d0d_characterizeImage_845930_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 190 has memoization hash b1abbbdcd778eecc9929af6c7c01b512 parsl.dataflow.memoization INFO: Task 190 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99d8040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 190 try 1 launched on executor multi with executor id 195 parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845930/f42fccef-f24c-44d0-83d0-c0b1dc938d0d_characterizeImage_845930_2.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845930/f42fccef-f24c-44d0-83d0-c0b1dc938d0d_characterizeImage_845930_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 192 try 0 failed parsl.dataflow.dflow INFO: Task 192 marked for retry parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845914/ebd9c59d-f745-4d03-a038-cf58a534897d_characterizeImage_845914_2.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845914/ebd9c59d-f745-4d03-a038-cf58a534897d_characterizeImage_845914_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 192 has memoization hash a376d0b9b6e26b25865f60ddcbedf788 parsl.dataflow.memoization INFO: Task 192 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99d80e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 192 try 1 launched on executor multi with executor id 196 parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845914/ebd9c59d-f745-4d03-a038-cf58a534897d_characterizeImage_845914_2.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845914/ebd9c59d-f745-4d03-a038-cf58a534897d_characterizeImage_845914_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 194 try 0 failed parsl.dataflow.dflow INFO: Task 194 marked for retry parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845974/12299a58-5e4e-409a-9e2e-f61ec6aea500_characterizeImage_845974_2.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845974/12299a58-5e4e-409a-9e2e-f61ec6aea500_characterizeImage_845974_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 194 has memoization hash 44cd58cd5c3776a198d2183f17aba522 parsl.dataflow.memoization INFO: Task 194 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99d84a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 194 try 1 launched on executor multi with executor id 197 parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845974/12299a58-5e4e-409a-9e2e-f61ec6aea500_characterizeImage_845974_2.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845974/12299a58-5e4e-409a-9e2e-f61ec6aea500_characterizeImage_845974_2.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 26, active_tasks = 104 parsl.dataflow.strategy DEBUG: Executor multi has 104 active tasks, 26/0 running/pending blocks, and 27 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23218217990800 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 196 try 0 failed parsl.dataflow.dflow INFO: Task 196 marked for retry parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845932/d3fd6454-1902-4c2b-84e1-0ae86cd08049_characterizeImage_845932_2.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845932/d3fd6454-1902-4c2b-84e1-0ae86cd08049_characterizeImage_845932_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 196 has memoization hash 734c3b65efd04261386a77579ffb4721 parsl.dataflow.memoization INFO: Task 196 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99d82c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 196 try 1 launched on executor multi with executor id 198 parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845932/d3fd6454-1902-4c2b-84e1-0ae86cd08049_characterizeImage_845932_2.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845932/d3fd6454-1902-4c2b-84e1-0ae86cd08049_characterizeImage_845932_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 198 try 0 failed parsl.dataflow.dflow INFO: Task 198 marked for retry parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845905/451fa2d1-c592-4332-b492-397ff2639846_characterizeImage_845905_2.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845905/451fa2d1-c592-4332-b492-397ff2639846_characterizeImage_845905_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 198 has memoization hash 5932264ca88347b2fcaaa05a168af8dd parsl.dataflow.memoization INFO: Task 198 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99d8680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 198 try 1 launched on executor multi with executor id 199 parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845905/451fa2d1-c592-4332-b492-397ff2639846_characterizeImage_845905_2.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845905/451fa2d1-c592-4332-b492-397ff2639846_characterizeImage_845905_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 200 try 0 failed parsl.dataflow.dflow INFO: Task 200 marked for retry parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845966/a7a8e70c-92d2-46fa-9a07-b1cec0342def_characterizeImage_845966_2.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845966/a7a8e70c-92d2-46fa-9a07-b1cec0342def_characterizeImage_845966_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 200 has memoization hash b95ed119de64950641922d71b294bf1e parsl.dataflow.memoization INFO: Task 200 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99d8860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 200 try 1 launched on executor multi with executor id 200 parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845966/a7a8e70c-92d2-46fa-9a07-b1cec0342def_characterizeImage_845966_2.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845966/a7a8e70c-92d2-46fa-9a07-b1cec0342def_characterizeImage_845966_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 202 try 0 failed parsl.dataflow.dflow INFO: Task 202 marked for retry parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845972/11cfd015-e4eb-41aa-a464-f15678e8e693_characterizeImage_845972_2.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845972/11cfd015-e4eb-41aa-a464-f15678e8e693_characterizeImage_845972_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 202 has memoization hash 2c2197ac36febbc47c3e875fc012069d parsl.dataflow.memoization INFO: Task 202 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99d8a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 202 try 1 launched on executor multi with executor id 201 parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845972/11cfd015-e4eb-41aa-a464-f15678e8e693_characterizeImage_845972_2.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845972/11cfd015-e4eb-41aa-a464-f15678e8e693_characterizeImage_845972_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 204 try 0 failed parsl.dataflow.dflow INFO: Task 204 marked for retry parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845877/a83d1524-ba7c-40ed-a158-6d1244bf3462_characterizeImage_845877_2.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845877/a83d1524-ba7c-40ed-a158-6d1244bf3462_characterizeImage_845877_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 204 has memoization hash b369fb9f690069e88e8d95a11e339689 parsl.dataflow.memoization INFO: Task 204 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99d8e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 204 try 1 launched on executor multi with executor id 202 parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845877/a83d1524-ba7c-40ed-a158-6d1244bf3462_characterizeImage_845877_2.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845877/a83d1524-ba7c-40ed-a158-6d1244bf3462_characterizeImage_845877_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 8 try 1 failed parsl.dataflow.dflow ERROR: Task 8 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 0a8648caf6226ed5ca02cccbe0a7329a with result from task 8 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 9 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 9 try 0 failed parsl.dataflow.dflow INFO: Task 9 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845879/278f6822-0d97-453f-b5b3-22cd38e70a08_calibrate_845879_2.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845879/278f6822-0d97-453f-b5b3-22cd38e70a08_calibrate_845879_2.stderr parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845879/9444aec8-7f5a-4bb7-abb6-e19832d51831_characterizeImage_845879_2.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845879/9444aec8-7f5a-4bb7-abb6-e19832d51831_characterizeImage_845879_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 6 try 1 failed parsl.dataflow.dflow ERROR: Task 6 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 88e67c74e0c27846fbc6159ad3abc379 with result from task 6 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 7 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 7 try 0 failed parsl.dataflow.dflow INFO: Task 7 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845917/23e35edf-28c6-4cc1-ab4a-e673791d337a_calibrate_845917_2.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845917/23e35edf-28c6-4cc1-ab4a-e673791d337a_calibrate_845917_2.stderr parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845917/6af4a453-83c9-42cf-b400-c9684629b977_characterizeImage_845917_2.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845917/6af4a453-83c9-42cf-b400-c9684629b977_characterizeImage_845917_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 206 try 0 failed parsl.dataflow.dflow INFO: Task 206 marked for retry parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845973/ebbf665d-168b-4cef-b190-f403cefe8c46_characterizeImage_845973_2.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845973/ebbf665d-168b-4cef-b190-f403cefe8c46_characterizeImage_845973_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 206 has memoization hash 414b1d50f04b6239e5666b74acba768d parsl.dataflow.memoization INFO: Task 206 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99dbd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 206 try 1 launched on executor multi with executor id 203 parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845973/ebbf665d-168b-4cef-b190-f403cefe8c46_characterizeImage_845973_2.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845973/ebbf665d-168b-4cef-b190-f403cefe8c46_characterizeImage_845973_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 12 try 1 failed parsl.dataflow.dflow ERROR: Task 12 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 98ac88c7618fcf1413ef0530b9ef8734 with result from task 12 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 13 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 13 try 0 failed parsl.dataflow.dflow INFO: Task 13 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845925/e31650f5-df3b-4dd5-9087-ccd70c82e662_calibrate_845925_2.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845925/e31650f5-df3b-4dd5-9087-ccd70c82e662_calibrate_845925_2.stderr parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845925/932d1e2c-a64f-4d9f-bf21-e4e5d4cff2fc_characterizeImage_845925_2.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845925/932d1e2c-a64f-4d9f-bf21-e4e5d4cff2fc_characterizeImage_845925_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 10 try 1 failed parsl.dataflow.dflow ERROR: Task 10 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 0e239dd7fa281aaaa938ba9801fea216 with result from task 10 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 11 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 11 try 0 failed parsl.dataflow.dflow INFO: Task 11 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845956/10745c5c-e424-4a47-879f-e70c3ceaf303_calibrate_845956_2.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845956/10745c5c-e424-4a47-879f-e70c3ceaf303_calibrate_845956_2.stderr parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845956/9dd128fe-62fa-4ea8-859e-c6327748c2b1_characterizeImage_845956_2.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845956/9dd128fe-62fa-4ea8-859e-c6327748c2b1_characterizeImage_845956_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 14 try 1 failed parsl.dataflow.dflow ERROR: Task 14 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 3159d685790ff3c487a08c113df38067 with result from task 14 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 15 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 15 try 0 failed parsl.dataflow.dflow INFO: Task 15 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845876/3114ac3d-5c4a-4d85-aaa8-2a55d89e2f20_calibrate_845876_2.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845876/3114ac3d-5c4a-4d85-aaa8-2a55d89e2f20_calibrate_845876_2.stderr parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845876/b79d0412-b23f-4fc3-9028-4c3becfc27d5_characterizeImage_845876_2.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845876/b79d0412-b23f-4fc3-9028-4c3becfc27d5_characterizeImage_845876_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 4 try 1 failed parsl.dataflow.dflow ERROR: Task 4 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 9d87fd5e64d591d853b837f65776411a with result from task 4 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 5 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 5 try 0 failed parsl.dataflow.dflow INFO: Task 5 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845923/f211a8d4-8336-4d16-b9ab-e46ad0c28904_calibrate_845923_2.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845923/f211a8d4-8336-4d16-b9ab-e46ad0c28904_calibrate_845923_2.stderr parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845923/307bb778-9554-4cf5-87a7-59a661125856_characterizeImage_845923_2.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845923/307bb778-9554-4cf5-87a7-59a661125856_characterizeImage_845923_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 48 try 1 failed parsl.dataflow.dflow ERROR: Task 48 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 017a71140c924a8d1120a2a44be86333 with result from task 48 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 49 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 49 try 0 failed parsl.dataflow.dflow INFO: Task 49 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845891/d4ce102c-7fa8-420f-9c7c-f24f4af04599_calibrate_845891_2.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845891/d4ce102c-7fa8-420f-9c7c-f24f4af04599_calibrate_845891_2.stderr parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845891/cd95ec31-d06c-4bdb-a6c3-642fc6d57370_characterizeImage_845891_2.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845891/cd95ec31-d06c-4bdb-a6c3-642fc6d57370_characterizeImage_845891_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 34 try 1 failed parsl.dataflow.dflow ERROR: Task 34 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry c01c083e32fcf85f4e300591a01b3000 with result from task 34 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 35 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 35 try 0 failed parsl.dataflow.dflow INFO: Task 35 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845938/9e727aac-fb63-4b76-b5f4-874823c0eb29_calibrate_845938_2.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845938/9e727aac-fb63-4b76-b5f4-874823c0eb29_calibrate_845938_2.stderr parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845938/04e8f597-23e1-4218-883d-b3fc7d9b8399_characterizeImage_845938_2.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845938/04e8f597-23e1-4218-883d-b3fc7d9b8399_characterizeImage_845938_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 42 try 1 failed parsl.dataflow.dflow ERROR: Task 42 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 7ba2785931ac668f5aa3083a3524de72 with result from task 42 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 43 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 43 try 0 failed parsl.dataflow.dflow INFO: Task 43 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845916/1071c667-6677-4eb3-9c10-f3a9e44e8a67_calibrate_845916_2.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845916/1071c667-6677-4eb3-9c10-f3a9e44e8a67_calibrate_845916_2.stderr parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845916/797024ed-829b-4ed2-9eaf-db520235adbf_characterizeImage_845916_2.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845916/797024ed-829b-4ed2-9eaf-db520235adbf_characterizeImage_845916_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 2 try 1 failed parsl.dataflow.dflow ERROR: Task 2 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 74965c9fb0108800374f15fc3f8aad54 with result from task 2 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 3 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 3 try 0 failed parsl.dataflow.dflow INFO: Task 3 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845969/c99c172f-3cdc-4a36-82e5-9774dc96b305_calibrate_845969_2.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845969/c99c172f-3cdc-4a36-82e5-9774dc96b305_calibrate_845969_2.stderr parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845969/e5463690-4c90-49b8-bf95-413e46148a08_characterizeImage_845969_2.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845969/e5463690-4c90-49b8-bf95-413e46148a08_characterizeImage_845969_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 32 try 1 failed parsl.dataflow.dflow ERROR: Task 32 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 469fb8ae935bab5e16cecb86e5f5d753 with result from task 32 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 33 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 33 try 0 failed parsl.dataflow.dflow INFO: Task 33 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845909/9f74af36-ac8b-4760-b7a8-0d9e4d847a14_calibrate_845909_2.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845909/9f74af36-ac8b-4760-b7a8-0d9e4d847a14_calibrate_845909_2.stderr parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845909/ac020887-2a2a-4cd3-973d-f2cf3d0ca195_characterizeImage_845909_2.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845909/ac020887-2a2a-4cd3-973d-f2cf3d0ca195_characterizeImage_845909_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 50 try 1 failed parsl.dataflow.dflow ERROR: Task 50 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry d04a3d9d8240c79d5986e53c79af23f9 with result from task 50 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 51 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 51 try 0 failed parsl.dataflow.dflow INFO: Task 51 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845953/44f206b1-a463-409c-bb6b-ee93794a37a6_calibrate_845953_2.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845953/44f206b1-a463-409c-bb6b-ee93794a37a6_calibrate_845953_2.stderr parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845953/2bd41c30-2e53-4ce5-9fa8-8eae6750494d_characterizeImage_845953_2.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845953/2bd41c30-2e53-4ce5-9fa8-8eae6750494d_characterizeImage_845953_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 0 try 1 failed parsl.dataflow.dflow ERROR: Task 0 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry dfcbb6858bd62a02b1c647f383fac018 with result from task 0 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 1 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 1 try 0 failed parsl.dataflow.dflow INFO: Task 1 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845889/ec3979bb-8558-475b-ae49-ab93da755014_calibrate_845889_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845889/ec3979bb-8558-475b-ae49-ab93da755014_calibrate_845889_2.stderr parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845889/47baa340-e987-4003-a9f7-a3eed82a19d4_characterizeImage_845889_2.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845889/47baa340-e987-4003-a9f7-a3eed82a19d4_characterizeImage_845889_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 40 try 1 failed parsl.dataflow.dflow ERROR: Task 40 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry fed050588955d9cdeb4e4959aef78ddf with result from task 40 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 41 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 41 try 0 failed parsl.dataflow.dflow INFO: Task 41 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845926/da46f4ad-1a6e-4be6-8a44-4bdb308533a3_calibrate_845926_2.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845926/da46f4ad-1a6e-4be6-8a44-4bdb308533a3_calibrate_845926_2.stderr parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845926/5a3176b1-cf38-4e9b-9585-03024c203f7e_characterizeImage_845926_2.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845926/5a3176b1-cf38-4e9b-9585-03024c203f7e_characterizeImage_845926_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 44 try 1 failed parsl.dataflow.dflow ERROR: Task 44 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 070ff849bee17ce25704e41606eee33c with result from task 44 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 45 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 45 try 0 failed parsl.dataflow.dflow INFO: Task 45 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845954/3033325d-744d-47e1-8eac-88251d033602_calibrate_845954_2.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845954/3033325d-744d-47e1-8eac-88251d033602_calibrate_845954_2.stderr parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845954/eaf6cc58-5cb9-4477-924a-e33e022dcc7f_characterizeImage_845954_2.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845954/eaf6cc58-5cb9-4477-924a-e33e022dcc7f_characterizeImage_845954_2.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 26, active_tasks = 89 parsl.dataflow.strategy DEBUG: Executor multi has 89 active tasks, 26/0 running/pending blocks, and 27 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23218217990800 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 38 try 1 failed parsl.dataflow.dflow ERROR: Task 38 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry f64e69ad7b5d5c9b4323b3d25b5f347f with result from task 38 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 39 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 39 try 0 failed parsl.dataflow.dflow INFO: Task 39 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845920/74dabd26-b33c-47ea-8464-f3c4d27dec03_calibrate_845920_2.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845920/74dabd26-b33c-47ea-8464-f3c4d27dec03_calibrate_845920_2.stderr parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845920/38b5da06-64ee-4e25-965b-c964097db327_characterizeImage_845920_2.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845920/38b5da06-64ee-4e25-965b-c964097db327_characterizeImage_845920_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 46 try 1 failed parsl.dataflow.dflow ERROR: Task 46 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry e890f1b1d6866bc6c740aa0eec7e066e with result from task 46 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 47 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 47 try 0 failed parsl.dataflow.dflow INFO: Task 47 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845886/8d165e1c-fc08-423f-b38c-2998f57dbb8a_calibrate_845886_2.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845886/8d165e1c-fc08-423f-b38c-2998f57dbb8a_calibrate_845886_2.stderr parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845886/9251d4db-ca14-412f-92e9-66fdd14198d5_characterizeImage_845886_2.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845886/9251d4db-ca14-412f-92e9-66fdd14198d5_characterizeImage_845886_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 24 try 1 failed parsl.dataflow.dflow ERROR: Task 24 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 18e2ca751e16fca96e811f0c522e28ba with result from task 24 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 25 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 25 try 0 failed parsl.dataflow.dflow INFO: Task 25 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845929/b6fe531b-2d86-48e5-8112-a5ae3e2519ae_calibrate_845929_2.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845929/b6fe531b-2d86-48e5-8112-a5ae3e2519ae_calibrate_845929_2.stderr parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845929/1a960c36-216f-4b57-b1f9-5a6150aadd7a_characterizeImage_845929_2.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845929/1a960c36-216f-4b57-b1f9-5a6150aadd7a_characterizeImage_845929_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 36 try 1 failed parsl.dataflow.dflow ERROR: Task 36 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry e5f44513b16c88757ba89a5eba595551 with result from task 36 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 37 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 37 try 0 failed parsl.dataflow.dflow INFO: Task 37 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845888/c30f225f-b8b8-4e6c-841a-f7ffcbd1bfaf_calibrate_845888_2.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845888/c30f225f-b8b8-4e6c-841a-f7ffcbd1bfaf_calibrate_845888_2.stderr parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845888/6662b1fa-0e7e-445c-94b7-c3e176b4f763_characterizeImage_845888_2.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845888/6662b1fa-0e7e-445c-94b7-c3e176b4f763_characterizeImage_845888_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 30 try 1 failed parsl.dataflow.dflow ERROR: Task 30 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry ba2ae0daa7158fc9112c89a875f03037 with result from task 30 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 31 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 31 try 0 failed parsl.dataflow.dflow INFO: Task 31 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845906/abf6e179-e765-4965-85ed-5cc22cb08f02_calibrate_845906_2.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845906/abf6e179-e765-4965-85ed-5cc22cb08f02_calibrate_845906_2.stderr parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845906/6892a379-1d64-46ab-9ec4-ce60e4af68dc_characterizeImage_845906_2.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845906/6892a379-1d64-46ab-9ec4-ce60e4af68dc_characterizeImage_845906_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 18 try 1 failed parsl.dataflow.dflow ERROR: Task 18 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry e996bdc0ebe1e7c59b4fc3cd6b3bd534 with result from task 18 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 19 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 19 try 0 failed parsl.dataflow.dflow INFO: Task 19 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845951/9d535b41-850c-49cf-8175-e7ab708b3811_calibrate_845951_2.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845951/9d535b41-850c-49cf-8175-e7ab708b3811_calibrate_845951_2.stderr parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845951/4cab1fe7-79b5-4f3d-8f17-0b8b0e1dea22_characterizeImage_845951_2.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845951/4cab1fe7-79b5-4f3d-8f17-0b8b0e1dea22_characterizeImage_845951_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 22 try 1 failed parsl.dataflow.dflow ERROR: Task 22 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry fbcfea2073e27e6eda147bae35d01516 with result from task 22 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 23 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 23 try 0 failed parsl.dataflow.dflow INFO: Task 23 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845962/0fd14cc7-e2c9-4d02-88c6-1ea0821d85fe_calibrate_845962_2.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845962/0fd14cc7-e2c9-4d02-88c6-1ea0821d85fe_calibrate_845962_2.stderr parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845962/9a3ed4f5-fec4-4bf8-80ff-9071e87fee48_characterizeImage_845962_2.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845962/9a3ed4f5-fec4-4bf8-80ff-9071e87fee48_characterizeImage_845962_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 16 try 1 failed parsl.dataflow.dflow ERROR: Task 16 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry cf4b320082b76b1e2fe672aab84cc0ec with result from task 16 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 17 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 17 try 0 failed parsl.dataflow.dflow INFO: Task 17 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845955/29eb2022-bf6c-4064-a209-440a48a429b1_calibrate_845955_2.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845955/29eb2022-bf6c-4064-a209-440a48a429b1_calibrate_845955_2.stderr parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845955/680a196a-21f7-494d-bc9c-ca5d4d30c999_characterizeImage_845955_2.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845955/680a196a-21f7-494d-bc9c-ca5d4d30c999_characterizeImage_845955_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 20 try 1 failed parsl.dataflow.dflow ERROR: Task 20 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 77ae5deac8c2a3a9f7fac6e9a5f42fbf with result from task 20 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 21 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 21 try 0 failed parsl.dataflow.dflow INFO: Task 21 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845949/28a95765-0c26-4e5b-a683-c1a14a007c11_calibrate_845949_2.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845949/28a95765-0c26-4e5b-a683-c1a14a007c11_calibrate_845949_2.stderr parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845949/f3355e5d-55be-4f4a-883f-ade688be4595_characterizeImage_845949_2.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845949/f3355e5d-55be-4f4a-883f-ade688be4595_characterizeImage_845949_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 28 try 1 failed parsl.dataflow.dflow ERROR: Task 28 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 719d7be68431e19edca9463945dbee24 with result from task 28 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 29 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 29 try 0 failed parsl.dataflow.dflow INFO: Task 29 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845895/dbab5775-80a9-487f-82da-8aa2cd020723_calibrate_845895_2.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845895/dbab5775-80a9-487f-82da-8aa2cd020723_calibrate_845895_2.stderr parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845895/c2b6501b-7148-4734-8d01-d113a76b8478_characterizeImage_845895_2.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845895/c2b6501b-7148-4734-8d01-d113a76b8478_characterizeImage_845895_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 26 try 1 failed parsl.dataflow.dflow ERROR: Task 26 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 048d028a81ac065b28de0c859d43adb6 with result from task 26 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 27 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 27 try 0 failed parsl.dataflow.dflow INFO: Task 27 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845924/6a4e1289-8891-4f6f-b296-42a382c4c452_calibrate_845924_2.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845924/6a4e1289-8891-4f6f-b296-42a382c4c452_calibrate_845924_2.stderr parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845924/ec22148e-7dcf-487c-85ce-66bade631426_characterizeImage_845924_2.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845924/ec22148e-7dcf-487c-85ce-66bade631426_characterizeImage_845924_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 86 try 1 failed parsl.dataflow.dflow ERROR: Task 86 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry bea4af1b9022478ce8f2cd1c66f0db0c with result from task 86 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 87 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 87 try 0 failed parsl.dataflow.dflow INFO: Task 87 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845936/250a5a5c-a7ba-42c4-8b13-3bc3a03ec8ea_calibrate_845936_2.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845936/250a5a5c-a7ba-42c4-8b13-3bc3a03ec8ea_calibrate_845936_2.stderr parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845936/0ad24b81-78f0-4a71-85b1-5897243e3959_characterizeImage_845936_2.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845936/0ad24b81-78f0-4a71-85b1-5897243e3959_characterizeImage_845936_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 60 try 1 failed parsl.dataflow.dflow ERROR: Task 60 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 251cbc3ea7dee02d57b87bb8c701f3c7 with result from task 60 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 61 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 61 try 0 failed parsl.dataflow.dflow INFO: Task 61 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845913/e4a20e55-f779-400f-95cd-9ef808c18cf0_calibrate_845913_2.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845913/e4a20e55-f779-400f-95cd-9ef808c18cf0_calibrate_845913_2.stderr parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845913/024ff1de-e31d-4139-b3c5-b73c055d82ca_characterizeImage_845913_2.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845913/024ff1de-e31d-4139-b3c5-b73c055d82ca_characterizeImage_845913_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 62 try 1 failed parsl.dataflow.dflow ERROR: Task 62 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry d8a281c1b3d9f765dac77891d60e22e5 with result from task 62 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 63 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 63 try 0 failed parsl.dataflow.dflow INFO: Task 63 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845939/45e6b42d-72e1-4c18-904e-d8e36b365c5f_calibrate_845939_2.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845939/45e6b42d-72e1-4c18-904e-d8e36b365c5f_calibrate_845939_2.stderr parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845939/b78c437e-0357-4a72-b7e7-e92ce8ff32e7_characterizeImage_845939_2.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845939/b78c437e-0357-4a72-b7e7-e92ce8ff32e7_characterizeImage_845939_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 58 try 1 failed parsl.dataflow.dflow ERROR: Task 58 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 16680ccd761e9f2b2599a7273b360be1 with result from task 58 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 59 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 59 try 0 failed parsl.dataflow.dflow INFO: Task 59 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845892/70936b88-ce64-41bc-a4aa-08f1aeea77d3_calibrate_845892_2.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845892/70936b88-ce64-41bc-a4aa-08f1aeea77d3_calibrate_845892_2.stderr parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845892/f31b5b11-e4a4-4d7d-a8a9-d6d25d84966c_characterizeImage_845892_2.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845892/f31b5b11-e4a4-4d7d-a8a9-d6d25d84966c_characterizeImage_845892_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 54 try 1 failed parsl.dataflow.dflow ERROR: Task 54 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry b5008116ffc61a6f3007ebe16fda163e with result from task 54 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 55 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 55 try 0 failed parsl.dataflow.dflow INFO: Task 55 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845927/a234c227-9268-4d29-a0a7-c1684eb56470_calibrate_845927_2.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845927/a234c227-9268-4d29-a0a7-c1684eb56470_calibrate_845927_2.stderr parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845927/86c93158-c0d4-4a62-8934-37cd77914d58_characterizeImage_845927_2.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845927/86c93158-c0d4-4a62-8934-37cd77914d58_characterizeImage_845927_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 52 try 1 failed parsl.dataflow.dflow ERROR: Task 52 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 58e47534fd0397325e9e00c6d86effed with result from task 52 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 53 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 53 try 0 failed parsl.dataflow.dflow INFO: Task 53 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845893/f53ee62c-55d6-48f7-898f-1488089cbedd_calibrate_845893_2.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845893/f53ee62c-55d6-48f7-898f-1488089cbedd_calibrate_845893_2.stderr parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845893/52c3beab-ecdd-4b0e-9db2-f6b12d2eb801_characterizeImage_845893_2.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845893/52c3beab-ecdd-4b0e-9db2-f6b12d2eb801_characterizeImage_845893_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 66 try 1 failed parsl.dataflow.dflow ERROR: Task 66 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 95b9654e07da090311c0189aef5b5d18 with result from task 66 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 26, active_tasks = 69 parsl.dataflow.strategy DEBUG: Executor multi has 69 active tasks, 26/0 running/pending blocks, and 27 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23218217990800 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 67 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 67 try 0 failed parsl.dataflow.dflow INFO: Task 67 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845975/8a0d1035-6086-4330-8f26-8d416a5394e6_calibrate_845975_2.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845975/8a0d1035-6086-4330-8f26-8d416a5394e6_calibrate_845975_2.stderr parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845975/da1c7d8f-e6f9-46a6-b3f2-8258eaeb2b51_characterizeImage_845975_2.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845975/da1c7d8f-e6f9-46a6-b3f2-8258eaeb2b51_characterizeImage_845975_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 64 try 1 failed parsl.dataflow.dflow ERROR: Task 64 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 292107de6482feee20e278a818f6538c with result from task 64 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 65 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 65 try 0 failed parsl.dataflow.dflow INFO: Task 65 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845874/4616e5b6-cabc-4801-aa5a-d2eb8b5bdcf8_calibrate_845874_2.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845874/4616e5b6-cabc-4801-aa5a-d2eb8b5bdcf8_calibrate_845874_2.stderr parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845874/4badbe55-5bf8-4c60-9092-cb226d85eda1_characterizeImage_845874_2.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845874/4badbe55-5bf8-4c60-9092-cb226d85eda1_characterizeImage_845874_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 56 try 1 failed parsl.dataflow.dflow ERROR: Task 56 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry f1f634f78f5dbf3f26226408456d4069 with result from task 56 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 57 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 57 try 0 failed parsl.dataflow.dflow INFO: Task 57 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845928/c9d31448-8218-4610-96e2-6679db741f33_calibrate_845928_2.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845928/c9d31448-8218-4610-96e2-6679db741f33_calibrate_845928_2.stderr parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845928/ea42931b-5e2c-48e0-908f-ad9a56071956_characterizeImage_845928_2.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845928/ea42931b-5e2c-48e0-908f-ad9a56071956_characterizeImage_845928_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 82 try 1 failed parsl.dataflow.dflow ERROR: Task 82 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 571deec6cb8c2b574c780b01457b7ed0 with result from task 82 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 83 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 83 try 0 failed parsl.dataflow.dflow INFO: Task 83 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845883/1ba16c68-b5e8-4906-ae21-0e1866d4e52e_calibrate_845883_2.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845883/1ba16c68-b5e8-4906-ae21-0e1866d4e52e_calibrate_845883_2.stderr parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845883/ea7a2c77-8df1-498a-aa9e-940279db3652_characterizeImage_845883_2.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845883/ea7a2c77-8df1-498a-aa9e-940279db3652_characterizeImage_845883_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 92 try 1 failed parsl.dataflow.dflow ERROR: Task 92 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 1a700a3cc267ef0bf4016ba03e69a613 with result from task 92 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 93 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 93 try 0 failed parsl.dataflow.dflow INFO: Task 93 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845907/fcea0716-8185-44dc-870f-8f9da82ef44e_calibrate_845907_2.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845907/fcea0716-8185-44dc-870f-8f9da82ef44e_calibrate_845907_2.stderr parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845907/cffcd057-7872-437d-9715-76380aa988d4_characterizeImage_845907_2.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845907/cffcd057-7872-437d-9715-76380aa988d4_characterizeImage_845907_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 70 try 1 failed parsl.dataflow.dflow ERROR: Task 70 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 59111f98da0a7bd3e808682dc15c3e52 with result from task 70 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 71 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 71 try 0 failed parsl.dataflow.dflow INFO: Task 71 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845967/f2264cb8-0305-486c-befe-d1024e99c3ef_calibrate_845967_2.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845967/f2264cb8-0305-486c-befe-d1024e99c3ef_calibrate_845967_2.stderr parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845967/5a555250-817e-4342-92e0-7aaa9746b09c_characterizeImage_845967_2.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845967/5a555250-817e-4342-92e0-7aaa9746b09c_characterizeImage_845967_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 68 try 1 failed parsl.dataflow.dflow ERROR: Task 68 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry d8ec5db9d7aa4d44b438da3d92317e06 with result from task 68 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 69 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 69 try 0 failed parsl.dataflow.dflow INFO: Task 69 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845945/de3fcf12-8e0f-40da-b630-05509d51a662_calibrate_845945_2.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845945/de3fcf12-8e0f-40da-b630-05509d51a662_calibrate_845945_2.stderr parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845945/3a266ace-a6aa-428e-88a7-b3fccd7d2042_characterizeImage_845945_2.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845945/3a266ace-a6aa-428e-88a7-b3fccd7d2042_characterizeImage_845945_2.stderr 2024-03-13 00:16:15 proc_lsst.multi:146 [INFO] found job 10228 in provider local proc_lsst.multi INFO: found job 10228 in provider local 2024-03-13 00:16:15 proc_lsst.multi:146 [INFO] found job 17006354 in provider astro proc_lsst.multi INFO: found job 17006354 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:15 proc_lsst.multi:146 [INFO] found job 17006355 in provider astro proc_lsst.multi INFO: found job 17006355 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 80 try 1 failed parsl.dataflow.dflow ERROR: Task 80 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry b09277256f23afd0bad289210c29570a with result from task 80 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 81 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 81 try 0 failed parsl.dataflow.dflow INFO: Task 81 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845957/55b98d6c-f810-48fd-b4ec-9211832c2ac9_calibrate_845957_2.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845957/55b98d6c-f810-48fd-b4ec-9211832c2ac9_calibrate_845957_2.stderr parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845957/7aa5b53f-f329-414b-9b47-d7b6f9e13666_characterizeImage_845957_2.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845957/7aa5b53f-f329-414b-9b47-d7b6f9e13666_characterizeImage_845957_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 76 try 1 failed parsl.dataflow.dflow ERROR: Task 76 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 6731d816e8d8239d3dbf0561c8423f22 with result from task 76 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 77 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 77 try 0 failed parsl.dataflow.dflow INFO: Task 77 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845885/1a4bfaf2-d324-433f-a9e2-55c06e31cfbd_calibrate_845885_2.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845885/1a4bfaf2-d324-433f-a9e2-55c06e31cfbd_calibrate_845885_2.stderr parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845885/af1fe9c9-54fc-473f-887d-3a7ae72aa8a3_characterizeImage_845885_2.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845885/af1fe9c9-54fc-473f-887d-3a7ae72aa8a3_characterizeImage_845885_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 74 try 1 failed parsl.dataflow.dflow ERROR: Task 74 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 54617f18a55d67a4158e9f7f5c124cce with result from task 74 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 75 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 75 try 0 failed parsl.dataflow.dflow INFO: Task 75 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845964/59bd08ee-77b1-4720-bbfd-ed8c5eaa48c3_calibrate_845964_2.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845964/59bd08ee-77b1-4720-bbfd-ed8c5eaa48c3_calibrate_845964_2.stderr parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845964/303de284-995e-41c2-8c35-f444be467aaa_characterizeImage_845964_2.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845964/303de284-995e-41c2-8c35-f444be467aaa_characterizeImage_845964_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:16 proc_lsst.multi:146 [INFO] found job 17006356 in provider astro proc_lsst.multi INFO: found job 17006356 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 88 try 1 failed parsl.dataflow.dflow ERROR: Task 88 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 41840f84335b33e2575117574502b150 with result from task 88 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 89 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 89 try 0 failed parsl.dataflow.dflow INFO: Task 89 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845947/32742f1a-c121-4df9-9f2d-ed4d9df82184_calibrate_845947_2.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845947/32742f1a-c121-4df9-9f2d-ed4d9df82184_calibrate_845947_2.stderr parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845947/f101aedf-9483-455c-b975-a91bbd2761c6_characterizeImage_845947_2.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845947/f101aedf-9483-455c-b975-a91bbd2761c6_characterizeImage_845947_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 72 try 1 failed parsl.dataflow.dflow ERROR: Task 72 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 78202a5dd2e162448f52d6908381c4c3 with result from task 72 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 73 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 73 try 0 failed parsl.dataflow.dflow INFO: Task 73 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845918/720c7013-5e46-4ddc-a224-0d681437ed3a_calibrate_845918_2.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845918/720c7013-5e46-4ddc-a224-0d681437ed3a_calibrate_845918_2.stderr parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845918/0b009a5e-e046-4261-8fa5-3c38569211ba_characterizeImage_845918_2.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845918/0b009a5e-e046-4261-8fa5-3c38569211ba_characterizeImage_845918_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 78 try 1 failed parsl.dataflow.dflow ERROR: Task 78 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry fe49a52810bd6bb64f918aabbddc662a with result from task 78 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 79 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 79 try 0 failed parsl.dataflow.dflow INFO: Task 79 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845941/d8e52196-8457-403c-97f6-0415993b73e8_calibrate_845941_2.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845941/d8e52196-8457-403c-97f6-0415993b73e8_calibrate_845941_2.stderr parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845941/0c7af297-2442-4256-b0b6-1c9c478a6c29_characterizeImage_845941_2.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845941/0c7af297-2442-4256-b0b6-1c9c478a6c29_characterizeImage_845941_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 84 try 1 failed parsl.dataflow.dflow ERROR: Task 84 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 5edcb9752b5fda47225a99f5e0a8d9c8 with result from task 84 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 85 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 85 try 0 failed parsl.dataflow.dflow INFO: Task 85 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845919/8de32a1b-18aa-40f1-8755-e79e59bea127_calibrate_845919_2.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845919/8de32a1b-18aa-40f1-8755-e79e59bea127_calibrate_845919_2.stderr parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845919/f5d0cb70-b110-4afa-b2a2-4a6f4f1450cf_characterizeImage_845919_2.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845919/f5d0cb70-b110-4afa-b2a2-4a6f4f1450cf_characterizeImage_845919_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 98 try 1 failed parsl.dataflow.dflow ERROR: Task 98 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry bba3b5a3efc393e35738df156d37271f with result from task 98 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 99 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 99 try 0 failed parsl.dataflow.dflow INFO: Task 99 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845958/f74d7a78-32d6-4ab8-975b-ff1420daaa10_calibrate_845958_2.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845958/f74d7a78-32d6-4ab8-975b-ff1420daaa10_calibrate_845958_2.stderr parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845958/d60f3d61-9a95-457d-97fe-f040390eb6f8_characterizeImage_845958_2.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845958/d60f3d61-9a95-457d-97fe-f040390eb6f8_characterizeImage_845958_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 100 try 1 failed parsl.dataflow.dflow ERROR: Task 100 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 2a005eae06a1644529e6f392ef4ef4c1 with result from task 100 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 101 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 101 try 0 failed parsl.dataflow.dflow INFO: Task 101 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845880/91530468-8963-4371-a6b1-1f8624cb3cf4_calibrate_845880_2.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845880/91530468-8963-4371-a6b1-1f8624cb3cf4_calibrate_845880_2.stderr parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845880/abd6192e-1c66-4408-9e4c-dec63d0b40ab_characterizeImage_845880_2.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845880/abd6192e-1c66-4408-9e4c-dec63d0b40ab_characterizeImage_845880_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 126 try 1 failed parsl.dataflow.dflow ERROR: Task 126 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 634b7baf979a5f735559d77d3a9ec9ca with result from task 126 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 127 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 127 try 0 failed parsl.dataflow.dflow INFO: Task 127 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845894/6656eff4-b816-4f6f-bfa7-d052e81607f8_calibrate_845894_2.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845894/6656eff4-b816-4f6f-bfa7-d052e81607f8_calibrate_845894_2.stderr parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845894/310d287a-0551-48bb-8f2d-6afe4b4d8367_characterizeImage_845894_2.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845894/310d287a-0551-48bb-8f2d-6afe4b4d8367_characterizeImage_845894_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 90 try 1 failed parsl.dataflow.dflow ERROR: Task 90 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 35cad0d2e8de02aa550181216d5111dc with result from task 90 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 91 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 91 try 0 failed parsl.dataflow.dflow INFO: Task 91 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845896/9b7db4a0-85a9-4ed0-bd40-cf4c15f460f5_calibrate_845896_2.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845896/9b7db4a0-85a9-4ed0-bd40-cf4c15f460f5_calibrate_845896_2.stderr parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845896/b046a7ec-89a9-45b3-ab8f-876092c79520_characterizeImage_845896_2.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845896/b046a7ec-89a9-45b3-ab8f-876092c79520_characterizeImage_845896_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 106 try 1 failed parsl.dataflow.dflow ERROR: Task 106 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 56f9b34c1d6d2fae87614ce08fb1d76a with result from task 106 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 107 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 107 try 0 failed parsl.dataflow.dflow INFO: Task 107 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845897/7a1e72ae-35ae-41de-9313-4ea6a4bf877a_calibrate_845897_2.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845897/7a1e72ae-35ae-41de-9313-4ea6a4bf877a_calibrate_845897_2.stderr parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845897/db4180e0-e7ad-4ce2-a1f2-4ce12f40d7d5_characterizeImage_845897_2.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845897/db4180e0-e7ad-4ce2-a1f2-4ce12f40d7d5_characterizeImage_845897_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 96 try 1 failed parsl.dataflow.dflow ERROR: Task 96 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry d970371a5a4b416dc608b556d1ed8ed2 with result from task 96 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 97 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 97 try 0 failed parsl.dataflow.dflow INFO: Task 97 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845908/451f54cb-6346-43e0-89dd-31e591a43f4d_calibrate_845908_2.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845908/451f54cb-6346-43e0-89dd-31e591a43f4d_calibrate_845908_2.stderr parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845908/2cccf77c-c5bd-400f-b9f6-e902f3419685_characterizeImage_845908_2.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845908/2cccf77c-c5bd-400f-b9f6-e902f3419685_characterizeImage_845908_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 104 try 1 failed parsl.dataflow.dflow ERROR: Task 104 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 26a197c55a718db18c438ed6ab2c43d0 with result from task 104 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 105 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 105 try 0 failed parsl.dataflow.dflow INFO: Task 105 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845946/c99f50ad-12fc-444a-811d-e027d049aa97_calibrate_845946_2.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845946/c99f50ad-12fc-444a-811d-e027d049aa97_calibrate_845946_2.stderr parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845946/1a01a87b-7ea3-4ff9-97de-18786cddcb4e_characterizeImage_845946_2.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845946/1a01a87b-7ea3-4ff9-97de-18786cddcb4e_characterizeImage_845946_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 124 try 1 failed parsl.dataflow.dflow ERROR: Task 124 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 28c8cdbabcc323c8e97ebf56b860f027 with result from task 124 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 125 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 125 try 0 failed parsl.dataflow.dflow INFO: Task 125 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845943/3f70a35a-ae84-4349-bd17-4c643ca60011_calibrate_845943_2.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845943/3f70a35a-ae84-4349-bd17-4c643ca60011_calibrate_845943_2.stderr parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845943/3d4b9af2-e295-4738-98ac-d7a871c68578_characterizeImage_845943_2.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845943/3d4b9af2-e295-4738-98ac-d7a871c68578_characterizeImage_845943_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 102 try 1 failed parsl.dataflow.dflow ERROR: Task 102 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry b17a8e6300c09938af380732f8a8f6e4 with result from task 102 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 103 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 103 try 0 failed parsl.dataflow.dflow INFO: Task 103 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845961/4acc1982-2bf7-4ad6-bb6b-919a4764a752_calibrate_845961_2.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845961/4acc1982-2bf7-4ad6-bb6b-919a4764a752_calibrate_845961_2.stderr parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845961/7727537b-7d07-45a6-9122-acbcb9e077ce_characterizeImage_845961_2.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845961/7727537b-7d07-45a6-9122-acbcb9e077ce_characterizeImage_845961_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:16 proc_lsst.multi:146 [INFO] found job 17006357 in provider astro proc_lsst.multi INFO: found job 17006357 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 94 try 1 failed parsl.dataflow.dflow ERROR: Task 94 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 1152693677a0369f82a87ee6d585cc04 with result from task 94 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 95 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 95 try 0 failed parsl.dataflow.dflow INFO: Task 95 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845899/9da62f9e-fe44-4c0d-9259-9b72e7f6b743_calibrate_845899_2.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845899/9da62f9e-fe44-4c0d-9259-9b72e7f6b743_calibrate_845899_2.stderr parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845899/56e7a245-664c-40c0-bd2b-24779e8ad2e0_characterizeImage_845899_2.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845899/56e7a245-664c-40c0-bd2b-24779e8ad2e0_characterizeImage_845899_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:17 proc_lsst.multi:146 [INFO] found job 17006358 in provider astro proc_lsst.multi INFO: found job 17006358 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 114 try 1 failed parsl.dataflow.dflow ERROR: Task 114 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 05e5a6aaca09e363ebff001ce54648f8 with result from task 114 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 115 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 115 try 0 failed parsl.dataflow.dflow INFO: Task 115 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845965/4889b597-68de-498f-b1f0-cf1a498d4aad_calibrate_845965_2.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845965/4889b597-68de-498f-b1f0-cf1a498d4aad_calibrate_845965_2.stderr parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845965/c6c0e87d-e3fc-4858-bef7-c19e9f1aea39_characterizeImage_845965_2.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845965/c6c0e87d-e3fc-4858-bef7-c19e9f1aea39_characterizeImage_845965_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:17 proc_lsst.multi:146 [INFO] found job 17006359 in provider astro proc_lsst.multi INFO: found job 17006359 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 128 try 1 failed parsl.dataflow.dflow ERROR: Task 128 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry da349ef6b94b0ceaf780a692c10ee37a with result from task 128 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 129 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 129 try 0 failed parsl.dataflow.dflow INFO: Task 129 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845902/ee77291b-8a73-4bd4-bae7-a42f4e09b94d_calibrate_845902_2.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845902/ee77291b-8a73-4bd4-bae7-a42f4e09b94d_calibrate_845902_2.stderr parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845902/33330bca-6bd1-4857-85da-515b4f803089_characterizeImage_845902_2.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845902/33330bca-6bd1-4857-85da-515b4f803089_characterizeImage_845902_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 138 try 1 failed parsl.dataflow.dflow ERROR: Task 138 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry a464cc6d36b1e40e30e451fcabdd05fb with result from task 138 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 139 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 139 try 0 failed parsl.dataflow.dflow INFO: Task 139 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845942/77b0c08e-2205-4ca0-90a3-31fd72f580e9_calibrate_845942_2.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845942/77b0c08e-2205-4ca0-90a3-31fd72f580e9_calibrate_845942_2.stderr parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845942/654704d2-6c3f-4dd2-a260-0a17b3b3d2ad_characterizeImage_845942_2.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845942/654704d2-6c3f-4dd2-a260-0a17b3b3d2ad_characterizeImage_845942_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:18 proc_lsst.multi:146 [INFO] found job 17006360 in provider astro proc_lsst.multi INFO: found job 17006360 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:18 proc_lsst.multi:146 [INFO] found job 17006361 in provider astro proc_lsst.multi INFO: found job 17006361 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:19 proc_lsst.multi:146 [INFO] found job 17006362 in provider astro proc_lsst.multi INFO: found job 17006362 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:19 proc_lsst.multi:146 [INFO] found job 17006363 in provider astro proc_lsst.multi INFO: found job 17006363 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:20 proc_lsst.multi:146 [INFO] found job 17006364 in provider astro proc_lsst.multi INFO: found job 17006364 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:20 proc_lsst.multi:146 [INFO] found job 17006365 in provider astro proc_lsst.multi INFO: found job 17006365 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:21 proc_lsst.multi:146 [INFO] found job 17006366 in provider astro proc_lsst.multi INFO: found job 17006366 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 134 try 1 failed parsl.dataflow.dflow ERROR: Task 134 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 4cb6affb9a82af1e1e6752ca4a88faf0 with result from task 134 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 135 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 135 try 0 failed parsl.dataflow.dflow INFO: Task 135 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845933/b67362f9-faa8-42c7-b920-bd7abd024475_calibrate_845933_2.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845933/b67362f9-faa8-42c7-b920-bd7abd024475_calibrate_845933_2.stderr parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845933/e24cc46c-28c5-4500-9354-c09907f1ab1a_characterizeImage_845933_2.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845933/e24cc46c-28c5-4500-9354-c09907f1ab1a_characterizeImage_845933_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:21 proc_lsst.multi:146 [INFO] found job 17006367 in provider astro proc_lsst.multi INFO: found job 17006367 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:22 proc_lsst.multi:146 [INFO] found job 17006368 in provider astro proc_lsst.multi INFO: found job 17006368 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:22 proc_lsst.multi:146 [INFO] found job 17006369 in provider astro proc_lsst.multi INFO: found job 17006369 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 118 try 1 failed parsl.dataflow.dflow ERROR: Task 118 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry f1502731263e78151067c062569940af with result from task 118 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 119 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 119 try 0 failed parsl.dataflow.dflow INFO: Task 119 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845921/1fdc543c-05bf-45bc-9c6f-247dd4b99b0f_calibrate_845921_2.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845921/1fdc543c-05bf-45bc-9c6f-247dd4b99b0f_calibrate_845921_2.stderr parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845921/5199ae07-c639-49d5-8195-de5cfcf37276_characterizeImage_845921_2.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845921/5199ae07-c639-49d5-8195-de5cfcf37276_characterizeImage_845921_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 116 try 1 failed parsl.dataflow.dflow ERROR: Task 116 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 58a5f1323cffab8aba0b198436379bbf with result from task 116 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 117 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 117 try 0 failed parsl.dataflow.dflow INFO: Task 117 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845960/64191299-3c76-418e-a689-40add6cd4fc3_calibrate_845960_2.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845960/64191299-3c76-418e-a689-40add6cd4fc3_calibrate_845960_2.stderr parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845960/0c21b0b0-2e4e-4feb-9471-2c58334fc3d3_characterizeImage_845960_2.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845960/0c21b0b0-2e4e-4feb-9471-2c58334fc3d3_characterizeImage_845960_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 130 try 1 failed parsl.dataflow.dflow ERROR: Task 130 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry d7977db724d22a2451ef2d979c1a6c27 with result from task 130 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 131 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 131 try 0 failed parsl.dataflow.dflow INFO: Task 131 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845915/e063cbc9-efa0-4ab8-896d-f7735167732a_calibrate_845915_2.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845915/e063cbc9-efa0-4ab8-896d-f7735167732a_calibrate_845915_2.stderr parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845915/6203f0d5-9054-4686-8b95-faef0a5edb06_characterizeImage_845915_2.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845915/6203f0d5-9054-4686-8b95-faef0a5edb06_characterizeImage_845915_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 132 try 1 failed parsl.dataflow.dflow ERROR: Task 132 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 5f1f9fea53aa5aeb4a8a20beb0ce6740 with result from task 132 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 133 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 133 try 0 failed parsl.dataflow.dflow INFO: Task 133 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845910/ad1622b3-6f6d-4bbc-b61e-447deff36090_calibrate_845910_2.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845910/ad1622b3-6f6d-4bbc-b61e-447deff36090_calibrate_845910_2.stderr parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845910/b6a1b982-bd05-4e4f-99e1-3adbfca0c6aa_characterizeImage_845910_2.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845910/b6a1b982-bd05-4e4f-99e1-3adbfca0c6aa_characterizeImage_845910_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 110 try 1 failed parsl.dataflow.dflow ERROR: Task 110 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 1b4d13183092be934a16bef0db0574f0 with result from task 110 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 111 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 111 try 0 failed parsl.dataflow.dflow INFO: Task 111 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845934/5964ee8f-fd0f-4a0f-ab77-fe54f79cbda0_calibrate_845934_2.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845934/5964ee8f-fd0f-4a0f-ab77-fe54f79cbda0_calibrate_845934_2.stderr parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845934/6d57084e-9f7b-4001-a687-2e72889d80fe_characterizeImage_845934_2.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845934/6d57084e-9f7b-4001-a687-2e72889d80fe_characterizeImage_845934_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 142 try 1 failed parsl.dataflow.dflow ERROR: Task 142 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 00b56b1de7658aa4bc1562e0d4c859fc with result from task 142 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 143 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 143 try 0 failed parsl.dataflow.dflow INFO: Task 143 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845971/eb9f807d-4b09-4f06-8c9e-3c554db10247_calibrate_845971_2.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845971/eb9f807d-4b09-4f06-8c9e-3c554db10247_calibrate_845971_2.stderr parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845971/85c8671d-b56e-4084-b9a7-e0c7fc16a71d_characterizeImage_845971_2.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845971/85c8671d-b56e-4084-b9a7-e0c7fc16a71d_characterizeImage_845971_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 108 try 1 failed parsl.dataflow.dflow ERROR: Task 108 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry b21401054395f5f8ba48adc9410115f2 with result from task 108 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 109 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 109 try 0 failed parsl.dataflow.dflow INFO: Task 109 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845890/6b1b947e-75e1-4cbc-a9d0-7c23d520e34d_calibrate_845890_2.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845890/6b1b947e-75e1-4cbc-a9d0-7c23d520e34d_calibrate_845890_2.stderr parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845890/2b57e739-0b5a-4f52-a6dd-18fe97769482_characterizeImage_845890_2.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845890/2b57e739-0b5a-4f52-a6dd-18fe97769482_characterizeImage_845890_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 122 try 1 failed parsl.dataflow.dflow ERROR: Task 122 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 8be815d32965502f9ba5eaf4f1f0e28c with result from task 122 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 123 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 123 try 0 failed parsl.dataflow.dflow INFO: Task 123 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:23 proc_lsst.multi:146 [INFO] found job 17006370 in provider astro proc_lsst.multi INFO: found job 17006370 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845931/c74355a6-c736-45a8-bf1d-0a5749cc0426_calibrate_845931_2.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845931/c74355a6-c736-45a8-bf1d-0a5749cc0426_calibrate_845931_2.stderr parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845931/24c0bb7d-7496-4719-a33c-c8419935b473_characterizeImage_845931_2.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845931/24c0bb7d-7496-4719-a33c-c8419935b473_characterizeImage_845931_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 146 try 1 failed parsl.dataflow.dflow ERROR: Task 146 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 8e4689bdd3a43f6074fc3be7d05ad7c0 with result from task 146 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 147 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 147 try 0 failed parsl.dataflow.dflow INFO: Task 147 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845887/28677bcd-cc7f-48de-80d3-f322fe105a71_calibrate_845887_2.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845887/28677bcd-cc7f-48de-80d3-f322fe105a71_calibrate_845887_2.stderr parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845887/1b07f960-0e34-4d29-bf40-2efafa1ec813_characterizeImage_845887_2.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845887/1b07f960-0e34-4d29-bf40-2efafa1ec813_characterizeImage_845887_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 140 try 1 failed parsl.dataflow.dflow ERROR: Task 140 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 2efcf3d6aaab8c7d58eab75596239da6 with result from task 140 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 141 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 141 try 0 failed parsl.dataflow.dflow INFO: Task 141 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845970/113169d6-33aa-4dc4-a043-0ce7f15d0f27_calibrate_845970_2.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845970/113169d6-33aa-4dc4-a043-0ce7f15d0f27_calibrate_845970_2.stderr parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845970/b5026a35-dc05-4010-b148-df5c3b5770b0_characterizeImage_845970_2.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845970/b5026a35-dc05-4010-b148-df5c3b5770b0_characterizeImage_845970_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 148 try 1 failed parsl.dataflow.dflow ERROR: Task 148 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry e291a612bb8e610169bc870795eca5dc with result from task 148 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 149 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 149 try 0 failed parsl.dataflow.dflow INFO: Task 149 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845937/6cc8fdcf-e459-431d-ae98-6db03cad325f_calibrate_845937_2.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845937/6cc8fdcf-e459-431d-ae98-6db03cad325f_calibrate_845937_2.stderr parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845937/ee634d28-bb24-424f-8807-5913155a88bb_characterizeImage_845937_2.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845937/ee634d28-bb24-424f-8807-5913155a88bb_characterizeImage_845937_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 144 try 1 failed parsl.dataflow.dflow ERROR: Task 144 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry af259f7fcd1ef5dfbc0433c91cc0a582 with result from task 144 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 145 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 145 try 0 failed parsl.dataflow.dflow INFO: Task 145 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845940/66fda4e3-6e34-4bac-b2ec-fdddf6de08c7_calibrate_845940_2.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845940/66fda4e3-6e34-4bac-b2ec-fdddf6de08c7_calibrate_845940_2.stderr parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845940/8242da42-4187-4773-a9dd-eaa55e453baa_characterizeImage_845940_2.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845940/8242da42-4187-4773-a9dd-eaa55e453baa_characterizeImage_845940_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 164 try 1 failed parsl.dataflow.dflow ERROR: Task 164 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry cbf9ce955b61128a561db8b181be4d0f with result from task 164 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 165 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 165 try 0 failed parsl.dataflow.dflow INFO: Task 165 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845898/ac682a27-b570-4a1c-83b5-e70f4d4c3313_calibrate_845898_2.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845898/ac682a27-b570-4a1c-83b5-e70f4d4c3313_calibrate_845898_2.stderr parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845898/28352836-5c66-4e8b-a3bb-a7128f2e6b3c_characterizeImage_845898_2.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845898/28352836-5c66-4e8b-a3bb-a7128f2e6b3c_characterizeImage_845898_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 120 try 1 failed parsl.dataflow.dflow ERROR: Task 120 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 39fa3e7c682bb37e7b54f33bc6e4fc55 with result from task 120 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 121 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 121 try 0 failed parsl.dataflow.dflow INFO: Task 121 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845944/65ff77f2-ac5a-409f-9b90-a3f14ce82056_calibrate_845944_2.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845944/65ff77f2-ac5a-409f-9b90-a3f14ce82056_calibrate_845944_2.stderr parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845944/a6044269-9d83-4d2b-b2c5-0d2f8fd837c1_characterizeImage_845944_2.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845944/a6044269-9d83-4d2b-b2c5-0d2f8fd837c1_characterizeImage_845944_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 150 try 1 failed parsl.dataflow.dflow ERROR: Task 150 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 9d9f8f84922b633fc8f11ababe429bd0 with result from task 150 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 151 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 151 try 0 failed parsl.dataflow.dflow INFO: Task 151 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845878/4627a2fc-8fc4-4c46-b453-b736c5a2b765_calibrate_845878_2.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845878/4627a2fc-8fc4-4c46-b453-b736c5a2b765_calibrate_845878_2.stderr parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845878/d5f15356-f090-4ae5-8e64-7f0360398e16_characterizeImage_845878_2.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845878/d5f15356-f090-4ae5-8e64-7f0360398e16_characterizeImage_845878_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 136 try 1 failed parsl.dataflow.dflow ERROR: Task 136 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry fa8677c0e62d828621eed9abfac2961e with result from task 136 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 137 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 137 try 0 failed parsl.dataflow.dflow INFO: Task 137 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845873/d9473702-f2f2-4deb-857c-04a4bd015a58_calibrate_845873_2.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845873/d9473702-f2f2-4deb-857c-04a4bd015a58_calibrate_845873_2.stderr parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845873/c4125fac-ad41-4766-890a-5eb55c5d1ff6_characterizeImage_845873_2.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845873/c4125fac-ad41-4766-890a-5eb55c5d1ff6_characterizeImage_845873_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 162 try 1 failed parsl.dataflow.dflow ERROR: Task 162 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 2f450815e21e44b313d3883b69ad19e3 with result from task 162 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 163 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 163 try 0 failed parsl.dataflow.dflow INFO: Task 163 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845903/97d1290e-1efc-436d-9b17-baf95476520c_calibrate_845903_2.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845903/97d1290e-1efc-436d-9b17-baf95476520c_calibrate_845903_2.stderr parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845903/da3e7f1f-c1f6-4f7b-a2f3-02a28506d989_characterizeImage_845903_2.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845903/da3e7f1f-c1f6-4f7b-a2f3-02a28506d989_characterizeImage_845903_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 112 try 1 failed parsl.dataflow.dflow ERROR: Task 112 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 2937485724674bc21461a4ffe8d183ff with result from task 112 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 113 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 113 try 0 failed parsl.dataflow.dflow INFO: Task 113 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845901/adc795f3-5bdf-44ed-8231-46f113c85223_calibrate_845901_2.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845901/adc795f3-5bdf-44ed-8231-46f113c85223_calibrate_845901_2.stderr parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845901/238275a5-1580-4fe5-874f-03519b7a35f4_characterizeImage_845901_2.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845901/238275a5-1580-4fe5-874f-03519b7a35f4_characterizeImage_845901_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:23 proc_lsst.multi:146 [INFO] found job 17006371 in provider astro proc_lsst.multi INFO: found job 17006371 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 182 try 1 failed parsl.dataflow.dflow ERROR: Task 182 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 83c32e90a036e41a982dc399a7d6f5af with result from task 182 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 183 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 183 try 0 failed parsl.dataflow.dflow INFO: Task 183 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845950/2b5893cb-d026-408b-ad5f-2aba47246c91_calibrate_845950_2.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845950/2b5893cb-d026-408b-ad5f-2aba47246c91_calibrate_845950_2.stderr parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845950/6f1a7668-a044-4ff6-b762-1a6e2bfc97f2_characterizeImage_845950_2.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845950/6f1a7668-a044-4ff6-b762-1a6e2bfc97f2_characterizeImage_845950_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:24 proc_lsst.multi:146 [INFO] found job 17006372 in provider astro proc_lsst.multi INFO: found job 17006372 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:24 proc_lsst.multi:146 [INFO] found job 17006373 in provider astro proc_lsst.multi INFO: found job 17006373 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 168 try 1 failed parsl.dataflow.dflow ERROR: Task 168 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 84f94359e378d4bd0ce4dbcf2dd93e4a with result from task 168 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 169 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 169 try 0 failed parsl.dataflow.dflow INFO: Task 169 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845875/7d0cdf00-a47f-4885-a36b-ab5bdfd48f0f_calibrate_845875_2.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845875/7d0cdf00-a47f-4885-a36b-ab5bdfd48f0f_calibrate_845875_2.stderr parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845875/9bc5aca3-17c2-45c9-9565-5995c7db1735_characterizeImage_845875_2.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845875/9bc5aca3-17c2-45c9-9565-5995c7db1735_characterizeImage_845875_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 178 try 1 failed parsl.dataflow.dflow ERROR: Task 178 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 28643aa54d498a59227ecef6d1e72523 with result from task 178 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 179 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 179 try 0 failed parsl.dataflow.dflow INFO: Task 179 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845900/d740a8d3-e40b-4b5c-9f26-f52f2679b10c_calibrate_845900_2.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845900/d740a8d3-e40b-4b5c-9f26-f52f2679b10c_calibrate_845900_2.stderr parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845900/488d80a1-d095-4def-b45a-12fc69abfa32_characterizeImage_845900_2.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845900/488d80a1-d095-4def-b45a-12fc69abfa32_characterizeImage_845900_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:25 proc_lsst.multi:146 [INFO] found job 17006374 in provider astro proc_lsst.multi INFO: found job 17006374 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:26 proc_lsst.multi:146 [INFO] found job 17006375 in provider astro proc_lsst.multi INFO: found job 17006375 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:26 proc_lsst.multi:146 [INFO] found job 17006376 in provider astro proc_lsst.multi INFO: found job 17006376 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:27 proc_lsst.multi:146 [INFO] found job 17006377 in provider astro proc_lsst.multi INFO: found job 17006377 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:27 proc_lsst.multi:146 [INFO] found job 17006378 in provider astro proc_lsst.multi INFO: found job 17006378 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 26, active_tasks = 23 parsl.dataflow.strategy DEBUG: Executor multi has 23 active tasks, 26/0 running/pending blocks, and 27 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23218217990800 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 26, active_tasks = 23 parsl.dataflow.strategy DEBUG: Executor multi has 23 active tasks, 26/0 running/pending blocks, and 27 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23218217990800 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 176 try 1 failed parsl.dataflow.dflow ERROR: Task 176 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 5f1bee9a3d168108278ed93378da91d3 with result from task 176 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 177 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 177 try 0 failed parsl.dataflow.dflow INFO: Task 177 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845872/eaf8962c-20ec-42ae-aadb-65e1128803a8_calibrate_845872_2.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845872/eaf8962c-20ec-42ae-aadb-65e1128803a8_calibrate_845872_2.stderr parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845872/f877b180-ff8a-4843-b8b3-6f68ec528560_characterizeImage_845872_2.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845872/f877b180-ff8a-4843-b8b3-6f68ec528560_characterizeImage_845872_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 174 try 1 failed parsl.dataflow.dflow ERROR: Task 174 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 795e44e2a3bf76947633282bb3375863 with result from task 174 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 175 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 175 try 0 failed parsl.dataflow.dflow INFO: Task 175 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845882/229af620-0cea-43b9-a728-2f452edf1f5b_calibrate_845882_2.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845882/229af620-0cea-43b9-a728-2f452edf1f5b_calibrate_845882_2.stderr parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845882/dd838f9a-4c46-46e4-a945-5f2a3d305458_characterizeImage_845882_2.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845882/dd838f9a-4c46-46e4-a945-5f2a3d305458_characterizeImage_845882_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 170 try 1 failed parsl.dataflow.dflow ERROR: Task 170 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry cf501f8bed69538cab5578a17074a0ee with result from task 170 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 171 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 171 try 0 failed parsl.dataflow.dflow INFO: Task 171 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845904/1168d62f-5d92-46e8-b367-be704258f254_calibrate_845904_2.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845904/1168d62f-5d92-46e8-b367-be704258f254_calibrate_845904_2.stderr parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845904/5f371e5b-014c-435d-bf1e-95319d9c08a8_characterizeImage_845904_2.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845904/5f371e5b-014c-435d-bf1e-95319d9c08a8_characterizeImage_845904_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 180 try 1 failed parsl.dataflow.dflow ERROR: Task 180 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry dfd9556d2e7a174dcdd1fbe33721d23e with result from task 180 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 181 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 181 try 0 failed parsl.dataflow.dflow INFO: Task 181 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845922/bea696a9-cf24-4350-b625-9f5f0133593e_calibrate_845922_2.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845922/bea696a9-cf24-4350-b625-9f5f0133593e_calibrate_845922_2.stderr parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845922/eb971c87-0a6c-4a04-9d00-74a1f966b764_characterizeImage_845922_2.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845922/eb971c87-0a6c-4a04-9d00-74a1f966b764_characterizeImage_845922_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 172 try 1 failed parsl.dataflow.dflow ERROR: Task 172 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry dbebc46a58a8c7b92a3991cf1867c43b with result from task 172 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 173 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 173 try 0 failed parsl.dataflow.dflow INFO: Task 173 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845881/5e6f5089-c206-4f47-a100-ffe5672e9393_calibrate_845881_2.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845881/5e6f5089-c206-4f47-a100-ffe5672e9393_calibrate_845881_2.stderr parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845881/1ef47dee-dced-4ac3-9577-1603c7c7b18a_characterizeImage_845881_2.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845881/1ef47dee-dced-4ac3-9577-1603c7c7b18a_characterizeImage_845881_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 186 try 1 failed parsl.dataflow.dflow ERROR: Task 186 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 785d72514708161c47089ddda2d9876b with result from task 186 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 187 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 187 try 0 failed parsl.dataflow.dflow INFO: Task 187 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845948/e8480088-96f4-4d6f-ad09-6e63f28e6339_calibrate_845948_2.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845948/e8480088-96f4-4d6f-ad09-6e63f28e6339_calibrate_845948_2.stderr parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845948/d09b2cb0-abb4-4627-a85a-84f4a7ef645a_characterizeImage_845948_2.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845948/d09b2cb0-abb4-4627-a85a-84f4a7ef645a_characterizeImage_845948_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 184 try 1 failed parsl.dataflow.dflow ERROR: Task 184 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 6be35a8648e1a0dc2874bb5e842da878 with result from task 184 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 185 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 185 try 0 failed parsl.dataflow.dflow INFO: Task 185 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845952/49234a6a-6b6e-4b68-9cc5-92ea91e9207f_calibrate_845952_2.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845952/49234a6a-6b6e-4b68-9cc5-92ea91e9207f_calibrate_845952_2.stderr parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845952/2d5dcdc4-3ad9-4a50-9879-e6bf6c638b35_characterizeImage_845952_2.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845952/2d5dcdc4-3ad9-4a50-9879-e6bf6c638b35_characterizeImage_845952_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 194 try 1 failed parsl.dataflow.dflow ERROR: Task 194 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 44cd58cd5c3776a198d2183f17aba522 with result from task 194 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 195 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 195 try 0 failed parsl.dataflow.dflow INFO: Task 195 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845974/413d125f-eb04-4138-a24e-568d888ce321_calibrate_845974_2.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845974/413d125f-eb04-4138-a24e-568d888ce321_calibrate_845974_2.stderr parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845974/12299a58-5e4e-409a-9e2e-f61ec6aea500_characterizeImage_845974_2.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845974/12299a58-5e4e-409a-9e2e-f61ec6aea500_characterizeImage_845974_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 192 try 1 failed parsl.dataflow.dflow ERROR: Task 192 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry a376d0b9b6e26b25865f60ddcbedf788 with result from task 192 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 193 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 193 try 0 failed parsl.dataflow.dflow INFO: Task 193 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845914/ae455b68-7a0a-4b1c-b8b4-bf6e0a3e1eec_calibrate_845914_2.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845914/ae455b68-7a0a-4b1c-b8b4-bf6e0a3e1eec_calibrate_845914_2.stderr parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845914/ebd9c59d-f745-4d03-a038-cf58a534897d_characterizeImage_845914_2.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845914/ebd9c59d-f745-4d03-a038-cf58a534897d_characterizeImage_845914_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 206 try 1 failed parsl.dataflow.dflow ERROR: Task 206 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 414b1d50f04b6239e5666b74acba768d with result from task 206 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 207 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 207 try 0 failed parsl.dataflow.dflow INFO: Task 207 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845973/044940f3-5f9a-4681-8871-fb17f6b8747b_calibrate_845973_2.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845973/044940f3-5f9a-4681-8871-fb17f6b8747b_calibrate_845973_2.stderr parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845973/ebbf665d-168b-4cef-b190-f403cefe8c46_characterizeImage_845973_2.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845973/ebbf665d-168b-4cef-b190-f403cefe8c46_characterizeImage_845973_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 196 try 1 failed parsl.dataflow.dflow ERROR: Task 196 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 734c3b65efd04261386a77579ffb4721 with result from task 196 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 197 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 197 try 0 failed parsl.dataflow.dflow INFO: Task 197 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845932/981cb7c4-978b-4087-9054-960c35ac6cf9_calibrate_845932_2.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845932/981cb7c4-978b-4087-9054-960c35ac6cf9_calibrate_845932_2.stderr parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845932/d3fd6454-1902-4c2b-84e1-0ae86cd08049_characterizeImage_845932_2.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845932/d3fd6454-1902-4c2b-84e1-0ae86cd08049_characterizeImage_845932_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 190 try 1 failed parsl.dataflow.dflow ERROR: Task 190 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry b1abbbdcd778eecc9929af6c7c01b512 with result from task 190 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 191 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 191 try 0 failed parsl.dataflow.dflow INFO: Task 191 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845930/0f1f5c52-e3f2-45c2-9df6-05238a829a77_calibrate_845930_2.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845930/0f1f5c52-e3f2-45c2-9df6-05238a829a77_calibrate_845930_2.stderr parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845930/f42fccef-f24c-44d0-83d0-c0b1dc938d0d_characterizeImage_845930_2.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845930/f42fccef-f24c-44d0-83d0-c0b1dc938d0d_characterizeImage_845930_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 202 try 1 failed parsl.dataflow.dflow ERROR: Task 202 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 2c2197ac36febbc47c3e875fc012069d with result from task 202 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 203 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 203 try 0 failed parsl.dataflow.dflow INFO: Task 203 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845972/78d13adc-8f12-4fc6-b1f3-b7fed3d03bae_calibrate_845972_2.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845972/78d13adc-8f12-4fc6-b1f3-b7fed3d03bae_calibrate_845972_2.stderr parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845972/11cfd015-e4eb-41aa-a464-f15678e8e693_characterizeImage_845972_2.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845972/11cfd015-e4eb-41aa-a464-f15678e8e693_characterizeImage_845972_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 198 try 1 failed parsl.dataflow.dflow ERROR: Task 198 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 5932264ca88347b2fcaaa05a168af8dd with result from task 198 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 199 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 199 try 0 failed parsl.dataflow.dflow INFO: Task 199 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845905/89ba4f54-f6b8-4af0-8748-5209e77e2a18_calibrate_845905_2.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845905/89ba4f54-f6b8-4af0-8748-5209e77e2a18_calibrate_845905_2.stderr parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845905/451fa2d1-c592-4332-b492-397ff2639846_characterizeImage_845905_2.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845905/451fa2d1-c592-4332-b492-397ff2639846_characterizeImage_845905_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 200 try 1 failed parsl.dataflow.dflow ERROR: Task 200 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry b95ed119de64950641922d71b294bf1e with result from task 200 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 201 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 201 try 0 failed parsl.dataflow.dflow INFO: Task 201 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845966/1c68cf06-5c95-48e9-bc1a-622a2ebf546a_calibrate_845966_2.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845966/1c68cf06-5c95-48e9-bc1a-622a2ebf546a_calibrate_845966_2.stderr parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845966/a7a8e70c-92d2-46fa-9a07-b1cec0342def_characterizeImage_845966_2.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845966/a7a8e70c-92d2-46fa-9a07-b1cec0342def_characterizeImage_845966_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 188 try 1 failed parsl.dataflow.dflow ERROR: Task 188 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry ab9ec5bc927e597b0f919b042eb5fe90 with result from task 188 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 189 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 189 try 0 failed parsl.dataflow.dflow INFO: Task 189 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845959/633f6545-95da-413a-b765-37b3b0b7daf3_calibrate_845959_2.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845959/633f6545-95da-413a-b765-37b3b0b7daf3_calibrate_845959_2.stderr parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845959/68a938ce-bc8c-4b0b-9e09-71045e95c554_characterizeImage_845959_2.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845959/68a938ce-bc8c-4b0b-9e09-71045e95c554_characterizeImage_845959_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 204 try 1 failed parsl.dataflow.dflow ERROR: Task 204 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry b369fb9f690069e88e8d95a11e339689 with result from task 204 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 205 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 205 try 0 failed parsl.dataflow.dflow INFO: Task 205 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845877/bfdf30f0-b7e1-4586-a0de-ae8d208ac387_calibrate_845877_2.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845877/bfdf30f0-b7e1-4586-a0de-ae8d208ac387_calibrate_845877_2.stderr parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845877/a83d1524-ba7c-40ed-a158-6d1244bf3462_characterizeImage_845877_2.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845877/a83d1524-ba7c-40ed-a158-6d1244bf3462_characterizeImage_845877_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 166 try 1 failed parsl.dataflow.dflow ERROR: Task 166 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 6018698b71a085912efb29d8c31fb68b with result from task 166 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 167 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 167 try 0 failed parsl.dataflow.dflow INFO: Task 167 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845884/f2de2e91-1d24-44d5-b193-2ef85b52e21a_calibrate_845884_2.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845884/f2de2e91-1d24-44d5-b193-2ef85b52e21a_calibrate_845884_2.stderr parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845884/cfcb4712-11f0-43a3-90d5-569ed4c00657_characterizeImage_845884_2.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845884/cfcb4712-11f0-43a3-90d5-569ed4c00657_characterizeImage_845884_2.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 26, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 26/0 running/pending blocks, and 27 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23218217990800 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 26, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 26/0 running/pending blocks, and 27 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23218217990800 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 26, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 26/0 running/pending blocks, and 27 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23218217990800 2024-03-13 00:16:48 proc_lsst.multi:146 [INFO] found job 10228 in provider local proc_lsst.multi INFO: found job 10228 in provider local 2024-03-13 00:16:48 proc_lsst.multi:146 [INFO] found job 17006354 in provider astro proc_lsst.multi INFO: found job 17006354 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:48 proc_lsst.multi:146 [INFO] found job 17006355 in provider astro proc_lsst.multi INFO: found job 17006355 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:48 proc_lsst.multi:146 [INFO] found job 17006356 in provider astro proc_lsst.multi INFO: found job 17006356 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:48 proc_lsst.multi:146 [INFO] found job 17006357 in provider astro proc_lsst.multi INFO: found job 17006357 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:48 proc_lsst.multi:146 [INFO] found job 17006358 in provider astro proc_lsst.multi INFO: found job 17006358 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:49 proc_lsst.multi:146 [INFO] found job 17006359 in provider astro proc_lsst.multi INFO: found job 17006359 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:49 proc_lsst.multi:146 [INFO] found job 17006360 in provider astro proc_lsst.multi INFO: found job 17006360 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:49 proc_lsst.multi:146 [INFO] found job 17006361 in provider astro proc_lsst.multi INFO: found job 17006361 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:49 proc_lsst.multi:146 [INFO] found job 17006362 in provider astro proc_lsst.multi INFO: found job 17006362 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:49 proc_lsst.multi:146 [INFO] found job 17006363 in provider astro proc_lsst.multi INFO: found job 17006363 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:49 proc_lsst.multi:146 [INFO] found job 17006364 in provider astro proc_lsst.multi INFO: found job 17006364 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:50 proc_lsst.multi:146 [INFO] found job 17006365 in provider astro proc_lsst.multi INFO: found job 17006365 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:50 proc_lsst.multi:146 [INFO] found job 17006366 in provider astro proc_lsst.multi INFO: found job 17006366 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:50 proc_lsst.multi:146 [INFO] found job 17006367 in provider astro proc_lsst.multi INFO: found job 17006367 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:50 proc_lsst.multi:146 [INFO] found job 17006368 in provider astro proc_lsst.multi INFO: found job 17006368 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:50 proc_lsst.multi:146 [INFO] found job 17006369 in provider astro proc_lsst.multi INFO: found job 17006369 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:50 proc_lsst.multi:146 [INFO] found job 17006370 in provider astro proc_lsst.multi INFO: found job 17006370 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:51 proc_lsst.multi:146 [INFO] found job 17006371 in provider astro proc_lsst.multi INFO: found job 17006371 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:51 proc_lsst.multi:146 [INFO] found job 17006372 in provider astro proc_lsst.multi INFO: found job 17006372 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:51 proc_lsst.multi:146 [INFO] found job 17006373 in provider astro proc_lsst.multi INFO: found job 17006373 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:51 proc_lsst.multi:146 [INFO] found job 17006374 in provider astro proc_lsst.multi INFO: found job 17006374 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:51 proc_lsst.multi:146 [INFO] found job 17006375 in provider astro proc_lsst.multi INFO: found job 17006375 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:51 proc_lsst.multi:146 [INFO] found job 17006376 in provider astro proc_lsst.multi INFO: found job 17006376 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:52 proc_lsst.multi:146 [INFO] found job 17006377 in provider astro proc_lsst.multi INFO: found job 17006377 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING 2024-03-13 00:16:52 proc_lsst.multi:146 [INFO] found job 17006378 in provider astro proc_lsst.multi INFO: found job 17006378 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006354,17006355,17006356,17006357,17006358,17006359,17006360,17006361,17006362,17006363,17006364,17006365,17006366,17006367,17006368,17006369,17006370,17006371,17006372,17006373,17006374,17006375,17006376,17006377,17006378' parsl.providers.slurm.slurm DEBUG: squeue returned 17006374 R 17006375 R 17006376 R 17006377 R 17006378 R 17006368 R 17006369 R 17006370 R 17006371 R 17006372 R 17006373 R 17006361 R 17006362 R 17006363 R 17006364 R 17006365 R 17006366 R 17006367 R 17006354 R 17006355 R 17006356 R 17006357 R 17006358 R 17006359 R 17006360 R parsl.providers.slurm.slurm DEBUG: Updating job 17006374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006378 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006360 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 26, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 26/0 running/pending blocks, and 27 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23218217990800 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 26, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 26/0 running/pending blocks, and 27 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23218217990800 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 26, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 26/0 running/pending blocks, and 27 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23218217990800 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 158 try 0 failed parsl.dataflow.dflow INFO: Task 158 marked for retry parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845963/70bff48a-f65e-4dcb-81d1-ac5f7b395120_characterizeImage_845963_2.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845963/70bff48a-f65e-4dcb-81d1-ac5f7b395120_characterizeImage_845963_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 158 has memoization hash d89618643baaf0770e9df36d002fbc8f parsl.dataflow.memoization INFO: Task 158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99960c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 158 try 1 launched on executor multi with executor id 204 parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845963/70bff48a-f65e-4dcb-81d1-ac5f7b395120_characterizeImage_845963_2.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845963/70bff48a-f65e-4dcb-81d1-ac5f7b395120_characterizeImage_845963_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 160 try 0 failed parsl.dataflow.dflow INFO: Task 160 marked for retry parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845968/64a13c94-f9ce-4b3f-81c7-49ecfa48fb85_characterizeImage_845968_2.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845968/64a13c94-f9ce-4b3f-81c7-49ecfa48fb85_characterizeImage_845968_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 160 has memoization hash 205471985a59a314bc1f41d58fb64027 parsl.dataflow.memoization INFO: Task 160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de99962a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 160 try 1 launched on executor multi with executor id 205 parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845968/64a13c94-f9ce-4b3f-81c7-49ecfa48fb85_characterizeImage_845968_2.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845968/64a13c94-f9ce-4b3f-81c7-49ecfa48fb85_characterizeImage_845968_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 152 try 0 failed parsl.dataflow.dflow INFO: Task 152 marked for retry parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845911/c26ae543-124e-43ed-9770-026445d465de_characterizeImage_845911_2.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845911/c26ae543-124e-43ed-9770-026445d465de_characterizeImage_845911_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 152 has memoization hash 87313c5ab2f030d7d5ba3ec97bcefa53 parsl.dataflow.memoization INFO: Task 152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9995b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 152 try 1 launched on executor multi with executor id 206 parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845911/c26ae543-124e-43ed-9770-026445d465de_characterizeImage_845911_2.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845911/c26ae543-124e-43ed-9770-026445d465de_characterizeImage_845911_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 154 try 0 failed parsl.dataflow.dflow INFO: Task 154 marked for retry parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845935/155c48c4-59fe-4597-9ead-47a1755574bb_characterizeImage_845935_2.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845935/155c48c4-59fe-4597-9ead-47a1755574bb_characterizeImage_845935_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 154 has memoization hash 7859859e4a8b3acfa0e25ffd22bd890e parsl.dataflow.memoization INFO: Task 154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9995ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 154 try 1 launched on executor multi with executor id 207 parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845935/155c48c4-59fe-4597-9ead-47a1755574bb_characterizeImage_845935_2.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845935/155c48c4-59fe-4597-9ead-47a1755574bb_characterizeImage_845935_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 156 try 0 failed parsl.dataflow.dflow INFO: Task 156 marked for retry parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845912/437c7ef3-b918-4ddb-bf93-6a9c10183898_characterizeImage_845912_2.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845912/437c7ef3-b918-4ddb-bf93-6a9c10183898_characterizeImage_845912_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 156 has memoization hash ccc2b86bd1c61d8b970774aae6f9d230 parsl.dataflow.memoization INFO: Task 156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x151de9995d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 156 try 1 launched on executor multi with executor id 208 parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845912/437c7ef3-b918-4ddb-bf93-6a9c10183898_characterizeImage_845912_2.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845912/437c7ef3-b918-4ddb-bf93-6a9c10183898_characterizeImage_845912_2.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 26, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 26/0 running/pending blocks, and 27 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23218217990800 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 158 try 1 failed parsl.dataflow.dflow ERROR: Task 158 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry d89618643baaf0770e9df36d002fbc8f with result from task 158 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 159 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 159 try 0 failed parsl.dataflow.dflow INFO: Task 159 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845963/5013a299-ff44-4503-b0b3-b7c3f1ec2ce1_calibrate_845963_2.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845963/5013a299-ff44-4503-b0b3-b7c3f1ec2ce1_calibrate_845963_2.stderr parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845963/70bff48a-f65e-4dcb-81d1-ac5f7b395120_characterizeImage_845963_2.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845963/70bff48a-f65e-4dcb-81d1-ac5f7b395120_characterizeImage_845963_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 152 try 1 failed parsl.dataflow.dflow ERROR: Task 152 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 87313c5ab2f030d7d5ba3ec97bcefa53 with result from task 152 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 153 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 153 try 0 failed parsl.dataflow.dflow INFO: Task 153 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845911/c3aedb25-5a2f-4201-bb56-5c11e9a1a29e_calibrate_845911_2.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845911/c3aedb25-5a2f-4201-bb56-5c11e9a1a29e_calibrate_845911_2.stderr parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845911/c26ae543-124e-43ed-9770-026445d465de_characterizeImage_845911_2.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845911/c26ae543-124e-43ed-9770-026445d465de_characterizeImage_845911_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 154 try 1 failed parsl.dataflow.dflow ERROR: Task 154 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 7859859e4a8b3acfa0e25ffd22bd890e with result from task 154 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 155 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 155 try 0 failed parsl.dataflow.dflow INFO: Task 155 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845935/106ec6a9-0930-4d95-a924-af070c96723b_calibrate_845935_2.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845935/106ec6a9-0930-4d95-a924-af070c96723b_calibrate_845935_2.stderr parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845935/155c48c4-59fe-4597-9ead-47a1755574bb_characterizeImage_845935_2.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845935/155c48c4-59fe-4597-9ead-47a1755574bb_characterizeImage_845935_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 160 try 1 failed parsl.dataflow.dflow ERROR: Task 160 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 205471985a59a314bc1f41d58fb64027 with result from task 160 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 161 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 161 try 0 failed parsl.dataflow.dflow INFO: Task 161 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845968/73e55331-8881-4ce5-bb14-a5ca63b20a7f_calibrate_845968_2.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845968/73e55331-8881-4ce5-bb14-a5ca63b20a7f_calibrate_845968_2.stderr parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845968/64a13c94-f9ce-4b3f-81c7-49ecfa48fb85_characterizeImage_845968_2.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845968/64a13c94-f9ce-4b3f-81c7-49ecfa48fb85_characterizeImage_845968_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 156 try 1 failed parsl.dataflow.dflow ERROR: Task 156 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry ccc2b86bd1c61d8b970774aae6f9d230 with result from task 156 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 157 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 157 try 0 failed parsl.dataflow.dflow INFO: Task 157 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow INFO: DFK cleanup initiated parsl.dataflow.dflow INFO: Summary of tasks in DFK: parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0 parsl.dataflow.dflow INFO: Tasks in state States.pending: 0 parsl.dataflow.dflow INFO: Tasks in state States.running: 0 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 0 parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845912/a45e3534-74b0-4115-92d4-78bd6b11870f_calibrate_845912_2.stdout parsl.dataflow.dflow INFO: Tasks in state States.failed: 104 parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/calibrate/845912/a45e3534-74b0-4115-92d4-78bd6b11870f_calibrate_845912_2.stderr parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 104 parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845912/437c7ef3-b918-4ddb-bf93-6a9c10183898_characterizeImage_845912_2.stdout parsl.dataflow.dflow INFO: Tasks in state States.launched: 0 parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/submit/DEEP/20190403/A0c/science#step1/20240313T001343Z/logs/characterizeImage/845912/437c7ef3-b918-4ddb-bf93-6a9c10183898_characterizeImage_845912_2.stderr parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0 parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0 parsl.dataflow.dflow INFO: Tasks in state States.joining: 0 parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0 parsl.dataflow.dflow INFO: End of summary parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Closing job status poller parsl.dataflow.dflow INFO: Terminated job status poller parsl.dataflow.dflow INFO: Scaling in and shutting down executors parsl.dataflow.dflow INFO: Scaling in executor multi parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=26, block_ids=[] parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 26 blocks parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cb87a7ec9098 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cb87a7ec9098 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8849a7251b0a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8849a7251b0a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dae57acf189d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dae57acf189d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ca9f80ebfd70 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ca9f80ebfd70 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a5bb3f120bf8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a5bb3f120bf8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dfa51f827b3a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dfa51f827b3a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 657728a7c314 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 657728a7c314 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ce146f1a32b2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ce146f1a32b2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 45da0c9abc5d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 45da0c9abc5d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5175a4c42fe4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5175a4c42fe4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3bd6a3e4b446 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3bd6a3e4b446 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4823c2da0c27 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4823c2da0c27 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0d2502d3ca6c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0d2502d3ca6c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6861378f95bd parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6861378f95bd parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e44a8af35333 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e44a8af35333 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 71995029a955 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 71995029a955 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5b82779264de parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5b82779264de parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 26b61b4d04e5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 26b61b4d04e5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2fa69b3f91f2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2fa69b3f91f2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 35ecfe56b36e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 35ecfe56b36e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fce4c4e40dfe parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fce4c4e40dfe parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 43226c0b81ee parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 43226c0b81ee parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 443ae37f6d08 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 443ae37f6d08 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5e5f9665fa27 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5e5f9665fa27 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 990ba5927339 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 990ba5927339 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 622ef0cf67f3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 622ef0cf67f3 2024-03-13 00:17:07 proc_lsst.multi:146 [INFO] found job 17006377 in provider astro proc_lsst.multi INFO: found job 17006377 in provider astro 2024-03-13 00:17:07 proc_lsst.multi:201 [INFO] cancelling 17006377 on provider astro proc_lsst.multi INFO: cancelling 17006377 on provider astro 2024-03-13 00:17:07 proc_lsst.multi:146 [INFO] found job 17006356 in provider astro proc_lsst.multi INFO: found job 17006356 in provider astro 2024-03-13 00:17:07 proc_lsst.multi:201 [INFO] cancelling 17006356 on provider astro proc_lsst.multi INFO: cancelling 17006356 on provider astro 2024-03-13 00:17:07 proc_lsst.multi:146 [INFO] found job 17006369 in provider astro proc_lsst.multi INFO: found job 17006369 in provider astro 2024-03-13 00:17:07 proc_lsst.multi:201 [INFO] cancelling 17006369 on provider astro proc_lsst.multi INFO: cancelling 17006369 on provider astro 2024-03-13 00:17:07 proc_lsst.multi:146 [INFO] found job 17006371 in provider astro proc_lsst.multi INFO: found job 17006371 in provider astro 2024-03-13 00:17:07 proc_lsst.multi:201 [INFO] cancelling 17006371 on provider astro proc_lsst.multi INFO: cancelling 17006371 on provider astro 2024-03-13 00:17:07 proc_lsst.multi:146 [INFO] found job 17006368 in provider astro proc_lsst.multi INFO: found job 17006368 in provider astro 2024-03-13 00:17:07 proc_lsst.multi:201 [INFO] cancelling 17006368 on provider astro proc_lsst.multi INFO: cancelling 17006368 on provider astro 2024-03-13 00:17:08 proc_lsst.multi:146 [INFO] found job 17006376 in provider astro proc_lsst.multi INFO: found job 17006376 in provider astro 2024-03-13 00:17:08 proc_lsst.multi:201 [INFO] cancelling 17006376 on provider astro proc_lsst.multi INFO: cancelling 17006376 on provider astro 2024-03-13 00:17:08 proc_lsst.multi:146 [INFO] found job 17006375 in provider astro proc_lsst.multi INFO: found job 17006375 in provider astro 2024-03-13 00:17:08 proc_lsst.multi:201 [INFO] cancelling 17006375 on provider astro proc_lsst.multi INFO: cancelling 17006375 on provider astro 2024-03-13 00:17:08 proc_lsst.multi:146 [INFO] found job 17006374 in provider astro proc_lsst.multi INFO: found job 17006374 in provider astro 2024-03-13 00:17:08 proc_lsst.multi:201 [INFO] cancelling 17006374 on provider astro proc_lsst.multi INFO: cancelling 17006374 on provider astro 2024-03-13 00:17:08 proc_lsst.multi:146 [INFO] found job 17006378 in provider astro proc_lsst.multi INFO: found job 17006378 in provider astro 2024-03-13 00:17:08 proc_lsst.multi:201 [INFO] cancelling 17006378 on provider astro proc_lsst.multi INFO: cancelling 17006378 on provider astro 2024-03-13 00:17:08 proc_lsst.multi:146 [INFO] found job 10228 in provider local proc_lsst.multi INFO: found job 10228 in provider local 2024-03-13 00:17:08 proc_lsst.multi:201 [INFO] cancelling 10228 on provider local proc_lsst.multi INFO: cancelling 10228 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 10228 2024-03-13 00:17:08 proc_lsst.multi:146 [INFO] found job 17006358 in provider astro proc_lsst.multi INFO: found job 17006358 in provider astro 2024-03-13 00:17:08 proc_lsst.multi:201 [INFO] cancelling 17006358 on provider astro proc_lsst.multi INFO: cancelling 17006358 on provider astro 2024-03-13 00:17:08 proc_lsst.multi:146 [INFO] found job 17006357 in provider astro proc_lsst.multi INFO: found job 17006357 in provider astro 2024-03-13 00:17:08 proc_lsst.multi:201 [INFO] cancelling 17006357 on provider astro proc_lsst.multi INFO: cancelling 17006357 on provider astro 2024-03-13 00:17:09 proc_lsst.multi:146 [INFO] found job 17006370 in provider astro proc_lsst.multi INFO: found job 17006370 in provider astro 2024-03-13 00:17:09 proc_lsst.multi:201 [INFO] cancelling 17006370 on provider astro proc_lsst.multi INFO: cancelling 17006370 on provider astro 2024-03-13 00:17:09 proc_lsst.multi:146 [INFO] found job 17006364 in provider astro proc_lsst.multi INFO: found job 17006364 in provider astro 2024-03-13 00:17:09 proc_lsst.multi:201 [INFO] cancelling 17006364 on provider astro proc_lsst.multi INFO: cancelling 17006364 on provider astro 2024-03-13 00:17:09 proc_lsst.multi:146 [INFO] found job 17006365 in provider astro proc_lsst.multi INFO: found job 17006365 in provider astro 2024-03-13 00:17:09 proc_lsst.multi:201 [INFO] cancelling 17006365 on provider astro proc_lsst.multi INFO: cancelling 17006365 on provider astro 2024-03-13 00:17:09 proc_lsst.multi:146 [INFO] found job 17006372 in provider astro proc_lsst.multi INFO: found job 17006372 in provider astro 2024-03-13 00:17:09 proc_lsst.multi:201 [INFO] cancelling 17006372 on provider astro proc_lsst.multi INFO: cancelling 17006372 on provider astro 2024-03-13 00:17:09 proc_lsst.multi:146 [INFO] found job 17006366 in provider astro proc_lsst.multi INFO: found job 17006366 in provider astro 2024-03-13 00:17:09 proc_lsst.multi:201 [INFO] cancelling 17006366 on provider astro proc_lsst.multi INFO: cancelling 17006366 on provider astro 2024-03-13 00:17:09 proc_lsst.multi:146 [INFO] found job 17006363 in provider astro proc_lsst.multi INFO: found job 17006363 in provider astro 2024-03-13 00:17:09 proc_lsst.multi:201 [INFO] cancelling 17006363 on provider astro proc_lsst.multi INFO: cancelling 17006363 on provider astro 2024-03-13 00:17:10 proc_lsst.multi:146 [INFO] found job 17006367 in provider astro proc_lsst.multi INFO: found job 17006367 in provider astro 2024-03-13 00:17:10 proc_lsst.multi:201 [INFO] cancelling 17006367 on provider astro proc_lsst.multi INFO: cancelling 17006367 on provider astro 2024-03-13 00:17:10 proc_lsst.multi:146 [INFO] found job 17006361 in provider astro proc_lsst.multi INFO: found job 17006361 in provider astro 2024-03-13 00:17:10 proc_lsst.multi:201 [INFO] cancelling 17006361 on provider astro proc_lsst.multi INFO: cancelling 17006361 on provider astro 2024-03-13 00:17:10 proc_lsst.multi:146 [INFO] found job 17006373 in provider astro proc_lsst.multi INFO: found job 17006373 in provider astro 2024-03-13 00:17:10 proc_lsst.multi:201 [INFO] cancelling 17006373 on provider astro proc_lsst.multi INFO: cancelling 17006373 on provider astro 2024-03-13 00:17:10 proc_lsst.multi:146 [INFO] found job 17006362 in provider astro proc_lsst.multi INFO: found job 17006362 in provider astro 2024-03-13 00:17:10 proc_lsst.multi:201 [INFO] cancelling 17006362 on provider astro proc_lsst.multi INFO: cancelling 17006362 on provider astro 2024-03-13 00:17:10 proc_lsst.multi:146 [INFO] found job 17006355 in provider astro proc_lsst.multi INFO: found job 17006355 in provider astro 2024-03-13 00:17:10 proc_lsst.multi:201 [INFO] cancelling 17006355 on provider astro proc_lsst.multi INFO: cancelling 17006355 on provider astro 2024-03-13 00:17:10 proc_lsst.multi:146 [INFO] found job 17006354 in provider astro proc_lsst.multi INFO: found job 17006354 in provider astro 2024-03-13 00:17:10 proc_lsst.multi:201 [INFO] cancelling 17006354 on provider astro proc_lsst.multi INFO: cancelling 17006354 on provider astro 2024-03-13 00:17:10 proc_lsst.multi:146 [INFO] found job 17006360 in provider astro proc_lsst.multi INFO: found job 17006360 in provider astro 2024-03-13 00:17:10 proc_lsst.multi:201 [INFO] cancelling 17006360 on provider astro proc_lsst.multi INFO: cancelling 17006360 on provider astro 2024-03-13 00:17:11 proc_lsst.multi:146 [INFO] found job 17006359 in provider astro proc_lsst.multi INFO: found job 17006359 in provider astro 2024-03-13 00:17:11 proc_lsst.multi:201 [INFO] cancelling 17006359 on provider astro proc_lsst.multi INFO: cancelling 17006359 on provider astro parsl.dataflow.dflow INFO: Shutting down executor multi 2024-03-13 00:17:11 proc_lsst.multi:40 [INFO] Cancelling all provider resources proc_lsst.multi INFO: Cancelling all provider resources 2024-03-13 00:17:11 proc_lsst.multi:47 [INFO] new jobs since last cancel ['17006354', '17006355', '17006356', '17006357', '17006358', '17006359', '17006360', '17006361', '17006362', '17006363', '17006364', '17006365', '17006366', '17006367', '17006368', '17006369', '17006370', '17006371', '17006372', '17006373', '17006374', '17006375', '17006376', '17006377', '17006378', '10228'] proc_lsst.multi INFO: new jobs since last cancel ['17006354', '17006355', '17006356', '17006357', '17006358', '17006359', '17006360', '17006361', '17006362', '17006363', '17006364', '17006365', '17006366', '17006367', '17006368', '17006369', '17006370', '17006371', '17006372', '17006373', '17006374', '17006375', '17006376', '17006377', '17006378', '10228'] 2024-03-13 00:17:11 proc_lsst.multi:146 [INFO] found job 17006354 in provider astro proc_lsst.multi INFO: found job 17006354 in provider astro 2024-03-13 00:17:11 proc_lsst.multi:201 [INFO] cancelling 17006354 on provider astro proc_lsst.multi INFO: cancelling 17006354 on provider astro 2024-03-13 00:17:11 proc_lsst.multi:146 [INFO] found job 17006355 in provider astro proc_lsst.multi INFO: found job 17006355 in provider astro 2024-03-13 00:17:11 proc_lsst.multi:201 [INFO] cancelling 17006355 on provider astro proc_lsst.multi INFO: cancelling 17006355 on provider astro 2024-03-13 00:17:11 proc_lsst.multi:146 [INFO] found job 17006356 in provider astro proc_lsst.multi INFO: found job 17006356 in provider astro 2024-03-13 00:17:11 proc_lsst.multi:201 [INFO] cancelling 17006356 on provider astro proc_lsst.multi INFO: cancelling 17006356 on provider astro 2024-03-13 00:17:11 proc_lsst.multi:146 [INFO] found job 17006357 in provider astro proc_lsst.multi INFO: found job 17006357 in provider astro 2024-03-13 00:17:11 proc_lsst.multi:201 [INFO] cancelling 17006357 on provider astro proc_lsst.multi INFO: cancelling 17006357 on provider astro 2024-03-13 00:17:11 proc_lsst.multi:146 [INFO] found job 17006358 in provider astro proc_lsst.multi INFO: found job 17006358 in provider astro 2024-03-13 00:17:11 proc_lsst.multi:201 [INFO] cancelling 17006358 on provider astro proc_lsst.multi INFO: cancelling 17006358 on provider astro 2024-03-13 00:17:12 proc_lsst.multi:146 [INFO] found job 17006359 in provider astro proc_lsst.multi INFO: found job 17006359 in provider astro 2024-03-13 00:17:12 proc_lsst.multi:201 [INFO] cancelling 17006359 on provider astro proc_lsst.multi INFO: cancelling 17006359 on provider astro 2024-03-13 00:17:12 proc_lsst.multi:146 [INFO] found job 17006360 in provider astro proc_lsst.multi INFO: found job 17006360 in provider astro 2024-03-13 00:17:12 proc_lsst.multi:201 [INFO] cancelling 17006360 on provider astro proc_lsst.multi INFO: cancelling 17006360 on provider astro 2024-03-13 00:17:12 proc_lsst.multi:146 [INFO] found job 17006361 in provider astro proc_lsst.multi INFO: found job 17006361 in provider astro 2024-03-13 00:17:12 proc_lsst.multi:201 [INFO] cancelling 17006361 on provider astro proc_lsst.multi INFO: cancelling 17006361 on provider astro 2024-03-13 00:17:12 proc_lsst.multi:146 [INFO] found job 17006362 in provider astro proc_lsst.multi INFO: found job 17006362 in provider astro 2024-03-13 00:17:12 proc_lsst.multi:201 [INFO] cancelling 17006362 on provider astro proc_lsst.multi INFO: cancelling 17006362 on provider astro 2024-03-13 00:17:12 proc_lsst.multi:146 [INFO] found job 17006363 in provider astro proc_lsst.multi INFO: found job 17006363 in provider astro 2024-03-13 00:17:12 proc_lsst.multi:201 [INFO] cancelling 17006363 on provider astro proc_lsst.multi INFO: cancelling 17006363 on provider astro 2024-03-13 00:17:12 proc_lsst.multi:146 [INFO] found job 17006364 in provider astro proc_lsst.multi INFO: found job 17006364 in provider astro 2024-03-13 00:17:12 proc_lsst.multi:201 [INFO] cancelling 17006364 on provider astro proc_lsst.multi INFO: cancelling 17006364 on provider astro 2024-03-13 00:17:12 proc_lsst.multi:146 [INFO] found job 17006365 in provider astro proc_lsst.multi INFO: found job 17006365 in provider astro 2024-03-13 00:17:12 proc_lsst.multi:201 [INFO] cancelling 17006365 on provider astro proc_lsst.multi INFO: cancelling 17006365 on provider astro 2024-03-13 00:17:13 proc_lsst.multi:146 [INFO] found job 17006366 in provider astro proc_lsst.multi INFO: found job 17006366 in provider astro 2024-03-13 00:17:13 proc_lsst.multi:201 [INFO] cancelling 17006366 on provider astro proc_lsst.multi INFO: cancelling 17006366 on provider astro 2024-03-13 00:17:13 proc_lsst.multi:146 [INFO] found job 17006367 in provider astro proc_lsst.multi INFO: found job 17006367 in provider astro 2024-03-13 00:17:13 proc_lsst.multi:201 [INFO] cancelling 17006367 on provider astro proc_lsst.multi INFO: cancelling 17006367 on provider astro 2024-03-13 00:17:13 proc_lsst.multi:146 [INFO] found job 17006368 in provider astro proc_lsst.multi INFO: found job 17006368 in provider astro 2024-03-13 00:17:13 proc_lsst.multi:201 [INFO] cancelling 17006368 on provider astro proc_lsst.multi INFO: cancelling 17006368 on provider astro 2024-03-13 00:17:13 proc_lsst.multi:146 [INFO] found job 17006369 in provider astro proc_lsst.multi INFO: found job 17006369 in provider astro 2024-03-13 00:17:13 proc_lsst.multi:201 [INFO] cancelling 17006369 on provider astro proc_lsst.multi INFO: cancelling 17006369 on provider astro 2024-03-13 00:17:13 proc_lsst.multi:146 [INFO] found job 17006370 in provider astro proc_lsst.multi INFO: found job 17006370 in provider astro 2024-03-13 00:17:13 proc_lsst.multi:201 [INFO] cancelling 17006370 on provider astro proc_lsst.multi INFO: cancelling 17006370 on provider astro 2024-03-13 00:17:13 proc_lsst.multi:146 [INFO] found job 17006371 in provider astro proc_lsst.multi INFO: found job 17006371 in provider astro 2024-03-13 00:17:13 proc_lsst.multi:201 [INFO] cancelling 17006371 on provider astro proc_lsst.multi INFO: cancelling 17006371 on provider astro 2024-03-13 00:17:14 proc_lsst.multi:146 [INFO] found job 17006372 in provider astro proc_lsst.multi INFO: found job 17006372 in provider astro 2024-03-13 00:17:14 proc_lsst.multi:201 [INFO] cancelling 17006372 on provider astro proc_lsst.multi INFO: cancelling 17006372 on provider astro 2024-03-13 00:17:14 proc_lsst.multi:146 [INFO] found job 17006373 in provider astro proc_lsst.multi INFO: found job 17006373 in provider astro 2024-03-13 00:17:14 proc_lsst.multi:201 [INFO] cancelling 17006373 on provider astro proc_lsst.multi INFO: cancelling 17006373 on provider astro 2024-03-13 00:17:14 proc_lsst.multi:146 [INFO] found job 17006374 in provider astro proc_lsst.multi INFO: found job 17006374 in provider astro 2024-03-13 00:17:14 proc_lsst.multi:201 [INFO] cancelling 17006374 on provider astro proc_lsst.multi INFO: cancelling 17006374 on provider astro 2024-03-13 00:17:14 proc_lsst.multi:146 [INFO] found job 17006375 in provider astro proc_lsst.multi INFO: found job 17006375 in provider astro 2024-03-13 00:17:14 proc_lsst.multi:201 [INFO] cancelling 17006375 on provider astro proc_lsst.multi INFO: cancelling 17006375 on provider astro 2024-03-13 00:17:14 proc_lsst.multi:146 [INFO] found job 17006376 in provider astro proc_lsst.multi INFO: found job 17006376 in provider astro 2024-03-13 00:17:14 proc_lsst.multi:201 [INFO] cancelling 17006376 on provider astro proc_lsst.multi INFO: cancelling 17006376 on provider astro 2024-03-13 00:17:14 proc_lsst.multi:146 [INFO] found job 17006377 in provider astro proc_lsst.multi INFO: found job 17006377 in provider astro 2024-03-13 00:17:14 proc_lsst.multi:201 [INFO] cancelling 17006377 on provider astro proc_lsst.multi INFO: cancelling 17006377 on provider astro 2024-03-13 00:17:14 proc_lsst.multi:146 [INFO] found job 17006378 in provider astro proc_lsst.multi INFO: found job 17006378 in provider astro 2024-03-13 00:17:14 proc_lsst.multi:201 [INFO] cancelling 17006378 on provider astro proc_lsst.multi INFO: cancelling 17006378 on provider astro 2024-03-13 00:17:15 proc_lsst.multi:146 [INFO] found job 10228 in provider local proc_lsst.multi INFO: found job 10228 in provider local 2024-03-13 00:17:15 proc_lsst.multi:201 [INFO] cancelling 10228 on provider local proc_lsst.multi INFO: cancelling 10228 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 10228 parsl.providers.local.local WARNING: Failed to kill PID: 10228 and child processes on local 2024-03-13 00:17:16 proc_lsst.multi:50 [INFO] no new jobs since last cancel, resuming executor shutdown proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt parsl.dataflow.dflow INFO: Shut down executor multi parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete parsl.executors.threads DEBUG: Done with executor shutdown parsl.dataflow.dflow INFO: Shut down executor _parsl_internal parsl.dataflow.dflow INFO: Terminated executors parsl.dataflow.dflow INFO: DFK cleanup complete parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 143.9787 seconds lsst.ctrl.bps.drivers INFO: Run 'DEEP_20190403_A0c_science#step1_20240313T001343Z' submitted for execution with id 'None' lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 143.9886 seconds; current memory usage: 0.344 Gibyte, delta: 0.003 Gibyte, peak delta: 0.003 Gibyte lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 161.2402 seconds; current memory usage: 0.344 Gibyte, delta: 0.207 Gibyte, peak delta: 0.207 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.344 Gibyte (main), 0.344 Gibyte (largest child process) Run Id: None Run Name: DEEP_20190403_A0c_science#step1_20240313T001343Z parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up real 2m44.199s user 0m44.678s sys 0m13.630s