+ PROC_LSST_SUBMIT_PATH=/mmfs1/home/stevengs/dirac/DEEP/submit + bps submit /gscratch/dirac/shared/opt/proc_lsst/pipelines/submit.yaml -b /mmfs1/home/stevengs/dirac/DEEP/repo -i DEEP/20190507/A1c --output-run DEEP/20190507/A1c/science#step1/20240313T014901Z --qgraph pipeline.qgraph lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process. lsst.ctrl.bps.drivers INFO: Starting submission process lsst.ctrl.bps.drivers INFO: Initializing execution environment lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 1.3749 seconds; current memory usage: 0.177 Gibyte, delta: 0.040 Gibyte, peak delta: 0.054 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.190 Gibyte (main), 0.000 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph) lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.0032 seconds lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.0063 seconds lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/pipeline_orig.qgraph DEEP/20190507/A1c/science#step1/20240313T014901Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/pipeline.qgraph lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 5.3625 seconds lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 13.1217 seconds; current memory usage: 0.343 Gibyte, delta: 0.167 Gibyte, peak delta: 0.153 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.343 Gibyte (main), 0.333 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs) lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0145 seconds; current memory usage: 0.343 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.343 Gibyte (main), 0.333 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 140 cluster(s) lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow) lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20190507_A1c_science#step1_20240313T014901Z' lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 0.0624 seconds; current memory usage: 0.343 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.343 Gibyte (main), 0.333 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 142 job(s) (including final) lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow) parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2 Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses s_addresses.add(address_by_interface(interface)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper retval = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface return socket.inet_ntoa(fcntl.ioctl( ^^^^^^^^^^^^ OSError: [Errno 99] Cannot assign requested address lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 0.1672 seconds; current memory usage: 0.345 Gibyte, delta: 0.001 Gibyte, peak delta: 0.001 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.345 Gibyte (main), 0.333 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting submit stage lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000 parsl.dataflow.dflow INFO: Starting DataFlowKernel with config Config( app_cache=True, checkpoint_files=None, checkpoint_mode='task_exit', checkpoint_period=None, executors=[MultiHighThroughputExecutor()], garbage_collect=True, initialize_logging=True, internal_tasks_max_threads=10, max_idletime=120.0, monitoring=None, retries=1, retry_handler=None, run_dir='runinfo', strategy='simple', usage_tracking=False ) parsl.dataflow.dflow INFO: Parsl version: 2023.06.12 parsl.usage_tracking.usage DEBUG: Tracking status: False parsl.dataflow.dflow INFO: Run id is: 9c101f7b-5552-4c89-bf97-990fe169010c parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name parsl.dataflow.memoization INFO: App caching initialized parsl.dataflow.strategy DEBUG: Scaling strategy: simple parsl.executors.high_throughput.executor DEBUG: Starting queue management thread parsl.executors.high_throughput.executor DEBUG: queue management worker starting parsl.executors.high_throughput.executor DEBUG: Started queue management thread Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z parsl.executors.high_throughput.executor DEBUG: Created management thread: parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id={block_id} --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider: parsl.executors.status_handling INFO: Scaling out by 1 blocks parsl.executors.status_handling INFO: Allocated block ID 0 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0 2024-03-13 01:49:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 2024-03-13 01:49:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 01:49:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-13 01:49:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 01:49:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local parsl.providers.local.local DEBUG: Launching in remote mode 2024-03-13 01:49:38 proc_lsst.multi:170 [INFO] [multi] job_id 23564 proc_lsst.multi INFO: [multi] job_id 23564 2024-03-13 01:49:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 01:49:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 01:49:38 proc_lsst.multi:178 [INFO] [multi] provider local accepted submit and returned 23564 proc_lsst.multi INFO: [multi] provider local accepted submit and returned 23564 parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 23564 parsl.dataflow.job_status_poller DEBUG: Adding executor multi parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 0 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 0 has memoization hash c877fe17b3afe9eaa5084e2b9088ce24 parsl.dataflow.memoization INFO: Task 0 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0210900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1 parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/19a20ea9-58db-4c2b-9e4e-82d1cf7cb8c9_calibrate_855911_35.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/19a20ea9-58db-4c2b-9e4e-82d1cf7cb8c9_calibrate_855911_35.stderr parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1 has memoization hash 945eb2730401d3236a5cb33f0270457a parsl.dataflow.memoization INFO: Task 1 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db02109a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 2 parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/846708ef-297c-4241-80c4-81276b622400_calibrate_855911_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/846708ef-297c-4241-80c4-81276b622400_calibrate_855911_9.stderr parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2 has memoization hash f4bf0c3c4eedcb763ff4ab113b3e09fe parsl.dataflow.memoization INFO: Task 2 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0210ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 3 parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/1ba9f290-426e-40fd-b776-5140f06d5856_calibrate_855911_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/1ba9f290-426e-40fd-b776-5140f06d5856_calibrate_855911_32.stderr parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3 has memoization hash 8060fa9cd95588ec1f90daf399058a28 parsl.dataflow.memoization INFO: Task 3 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0210f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 4 parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/fc6e7088-7176-4422-96f7-861be6218102_calibrate_855911_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/fc6e7088-7176-4422-96f7-861be6218102_calibrate_855911_14.stderr parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash f5dc0abcdf5fdb8a5a6b8ffcfc7941a0 parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0210d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 5 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/64c54979-dfdb-429a-897d-cbd231b11b96_calibrate_855911_59.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/64c54979-dfdb-429a-897d-cbd231b11b96_calibrate_855911_59.stderr parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5 has memoization hash 7b9f97e5556c134f7b40ed88f2184075 parsl.dataflow.memoization INFO: Task 5 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0210fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 6 parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/db2895de-bf84-4a3e-8579-ea13eb770871_calibrate_855911_34.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/db2895de-bf84-4a3e-8579-ea13eb770871_calibrate_855911_34.stderr parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 6 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 6 has memoization hash d6db8b96e78af2a4f34530c3c84023ad parsl.dataflow.memoization INFO: Task 6 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0211080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 7 parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/634fa457-accc-4576-90ce-3d577bc12af2_calibrate_855911_25.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/634fa457-accc-4576-90ce-3d577bc12af2_calibrate_855911_25.stderr parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 7 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 7 has memoization hash a4de438e871dead9fe871ac68864ef3c parsl.dataflow.memoization INFO: Task 7 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0210220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 8 parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/235a9997-cb8d-4b9d-927a-026f2e0b49dd_calibrate_855911_41.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/235a9997-cb8d-4b9d-927a-026f2e0b49dd_calibrate_855911_41.stderr parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 8 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 8 has memoization hash 5be10890c0174fc8f64aad9dfabce424 parsl.dataflow.memoization INFO: Task 8 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0210a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 9 parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/63247ced-0399-4767-bf05-e6d64abbfb8d_calibrate_855910_9.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/63247ced-0399-4767-bf05-e6d64abbfb8d_calibrate_855910_9.stderr parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 9 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 9 has memoization hash 62215dcfe5e4e0955f0641f249b0e10d parsl.dataflow.memoization INFO: Task 9 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0211120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 10 parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/8b2fa882-73b1-4798-923c-386c396bac28_calibrate_855911_27.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/8b2fa882-73b1-4798-923c-386c396bac28_calibrate_855911_27.stderr parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 10 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash 569c5fab366f4d0989df9b03d78cace5 parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0210e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 11 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/89d5f5c0-6cde-42f9-9981-aa0a6946ff66_calibrate_855910_34.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/89d5f5c0-6cde-42f9-9981-aa0a6946ff66_calibrate_855910_34.stderr parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 11 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 11 has memoization hash 1f0a16be7fea9254b03f305974f9a281 parsl.dataflow.memoization INFO: Task 11 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0211300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 12 parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/13eadaed-b6ed-46a9-a1fe-1e9790987a07_calibrate_855910_21.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/13eadaed-b6ed-46a9-a1fe-1e9790987a07_calibrate_855910_21.stderr parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 12 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 12 has memoization hash b7bcf3fad9715cc4965aac2677681012 parsl.dataflow.memoization INFO: Task 12 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0211440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 13 parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/d136b04a-8eac-4050-8d12-d225ce3c1cf0_calibrate_855910_35.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/d136b04a-8eac-4050-8d12-d225ce3c1cf0_calibrate_855910_35.stderr parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 13 submitted for App calibrate, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 13 has memoization hash e4cdb6332ecaf44719ea625b51e0c07e parsl.dataflow.memoization INFO: Task 13 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0211800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 14 parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/8784752d-bf08-47df-b8a0-c97ff7f9b4a0_calibrate_855910_4.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/8784752d-bf08-47df-b8a0-c97ff7f9b4a0_calibrate_855910_4.stderr parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 14 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 14 has memoization hash 5d5e14c03ea0cf657543b8fa137794fd parsl.dataflow.memoization INFO: Task 14 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0211940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 15 parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/08082892-9973-49b4-81e5-e3b332deac5b_characterizeImage_855911_38.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/08082892-9973-49b4-81e5-e3b332deac5b_characterizeImage_855911_38.stderr parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 15 submitted for App calibrate, waiting on task 14 parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 15 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 16 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash f919b0be28762e72b432c44f5167d861 parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0211760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 16 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/9390e644-3687-46b6-aa37-eb7c80d4de29_characterizeImage_855910_16.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/9390e644-3687-46b6-aa37-eb7c80d4de29_characterizeImage_855910_16.stderr parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 17 submitted for App calibrate, waiting on task 16 parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 17 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 18 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 18 has memoization hash bb34b92b8d6ed10882635aeefc65760a parsl.dataflow.memoization INFO: Task 18 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0211a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 17 parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/a090aba4-4a38-4573-a572-f5cabab5b056_characterizeImage_855911_13.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/a090aba4-4a38-4573-a572-f5cabab5b056_characterizeImage_855911_13.stderr parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 19 submitted for App calibrate, waiting on task 18 parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 19 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 20 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 20 has memoization hash 7cdaea0c83a308f4fe22ba2e71c4ee16 parsl.dataflow.memoization INFO: Task 20 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0211c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 18 parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/ca786ff8-5ae7-4c4c-845b-c815ba80c3a2_characterizeImage_855911_15.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/ca786ff8-5ae7-4c4c-845b-c815ba80c3a2_characterizeImage_855911_15.stderr parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 21 submitted for App calibrate, waiting on task 20 parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 21 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 22 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash 9f73ec5b9a7aef9a54c21b04299c3355 parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0211f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 19 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/800a2922-f721-4d25-822c-a7094ee8caed_characterizeImage_855910_46.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/800a2922-f721-4d25-822c-a7094ee8caed_characterizeImage_855910_46.stderr parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 23 submitted for App calibrate, waiting on task 22 parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 23 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 24 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 24 has memoization hash 843374210022ec35e6ac2113ed5276ce parsl.dataflow.memoization INFO: Task 24 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0212160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 20 parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/6175499a-5177-4137-b42d-38fc1c31a263_characterizeImage_855911_58.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/6175499a-5177-4137-b42d-38fc1c31a263_characterizeImage_855911_58.stderr parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 25 submitted for App calibrate, waiting on task 24 parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 25 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 26 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 26 has memoization hash 1470caf28784b09d64c5bf305806b824 parsl.dataflow.memoization INFO: Task 26 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0212520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 21 parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/1a97605c-6432-4995-b290-68cb12b763a2_characterizeImage_855911_5.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/1a97605c-6432-4995-b290-68cb12b763a2_characterizeImage_855911_5.stderr parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 27 submitted for App calibrate, waiting on task 26 parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 27 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 28 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash 8aafad68ee30a513d7ff0fc7ce1f5c26 parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db02120c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 22 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/983a01f8-c7bf-4fef-84c8-a8caf58c74b6_characterizeImage_855911_39.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/983a01f8-c7bf-4fef-84c8-a8caf58c74b6_characterizeImage_855911_39.stderr parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 29 submitted for App calibrate, waiting on task 28 parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 29 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 30 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 30 has memoization hash c8f9f8853974013de48d830f921f325c parsl.dataflow.memoization INFO: Task 30 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0212660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 23 parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/5c0d4d51-2440-430b-9231-d536d662efe7_characterizeImage_855911_11.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/5c0d4d51-2440-430b-9231-d536d662efe7_characterizeImage_855911_11.stderr parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 31 submitted for App calibrate, waiting on task 30 parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 31 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 32 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 32 has memoization hash a38cb225b314a6deb4f26a38c497b849 parsl.dataflow.memoization INFO: Task 32 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0213a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 24 parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/8d6c5ccf-e869-40b5-8751-c8d8d42c408d_characterizeImage_855910_27.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/8d6c5ccf-e869-40b5-8751-c8d8d42c408d_characterizeImage_855910_27.stderr parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 33 submitted for App calibrate, waiting on task 32 parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 33 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 34 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash 075cfe43ceab10b0e0991f5e1ce3038b parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0213920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 25 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/4f91a3bb-dde6-45ad-ae80-2cec03be0edf_characterizeImage_855911_16.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/4f91a3bb-dde6-45ad-ae80-2cec03be0edf_characterizeImage_855911_16.stderr parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 35 submitted for App calibrate, waiting on task 34 parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 35 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 36 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 36 has memoization hash 3e33fcbdba0efe15fcc1181bee9baaaf parsl.dataflow.memoization INFO: Task 36 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0213560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 26 parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/256f3f22-a5e3-4dc6-bfbf-67a9cd648029_characterizeImage_855911_21.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/256f3f22-a5e3-4dc6-bfbf-67a9cd648029_characterizeImage_855911_21.stderr parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 37 submitted for App calibrate, waiting on task 36 parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 37 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 38 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 38 has memoization hash 279fc759a984b5b6d09ebe68191901a9 parsl.dataflow.memoization INFO: Task 38 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0213740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 27 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/182ff692-3216-4bc9-a828-1736d1daf016_characterizeImage_855911_1.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/182ff692-3216-4bc9-a828-1736d1daf016_characterizeImage_855911_1.stderr parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 39 submitted for App calibrate, waiting on task 38 parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 39 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 40 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash 695315fd2af3281397c8bf1391395ac0 parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0213380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 28 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/c82fb047-b7dc-49ad-845d-7d9c7cc1ecd1_characterizeImage_855910_47.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/c82fb047-b7dc-49ad-845d-7d9c7cc1ecd1_characterizeImage_855910_47.stderr parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 41 submitted for App calibrate, waiting on task 40 parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 41 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 42 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 42 has memoization hash dd11e2e4560735a94dba9365e9b48f75 parsl.dataflow.memoization INFO: Task 42 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db02131a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 29 parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/8774d934-8e13-4ee7-9af4-b8633afb8117_characterizeImage_855911_3.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/8774d934-8e13-4ee7-9af4-b8633afb8117_characterizeImage_855911_3.stderr parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 43 submitted for App calibrate, waiting on task 42 parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 43 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 44 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 44 has memoization hash 3bff9b078737f4a7145ba0c4f60a220c parsl.dataflow.memoization INFO: Task 44 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0212fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 30 parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/22235225-dfff-445b-b762-eec54521fba4_characterizeImage_855910_51.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/22235225-dfff-445b-b762-eec54521fba4_characterizeImage_855910_51.stderr parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 45 submitted for App calibrate, waiting on task 44 parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 45 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 46 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 46 has memoization hash ef41369a22e73637504962f6c442c39b parsl.dataflow.memoization INFO: Task 46 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0212c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 31 parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/640eda07-0934-42c3-a791-049094bd785a_characterizeImage_855911_22.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/640eda07-0934-42c3-a791-049094bd785a_characterizeImage_855911_22.stderr parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 47 submitted for App calibrate, waiting on task 46 parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 47 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 48 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 48 has memoization hash 7ee445c26b0e81f76caf54fd1e2daa1c parsl.dataflow.memoization INFO: Task 48 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0212de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 32 parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855909/d4097b91-b153-43b0-866d-30e0c77604e5_characterizeImage_855909_2.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855909/d4097b91-b153-43b0-866d-30e0c77604e5_characterizeImage_855909_2.stderr parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 49 submitted for App calibrate, waiting on task 48 parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 49 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 50 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 50 has memoization hash c81f729b57ed59987b5c33c48e4370ad parsl.dataflow.memoization INFO: Task 50 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0212a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 33 parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/0a7de8f6-9d73-4bcc-9593-33726fcaba42_characterizeImage_855911_48.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/0a7de8f6-9d73-4bcc-9593-33726fcaba42_characterizeImage_855911_48.stderr parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 51 submitted for App calibrate, waiting on task 50 parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 51 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 52 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash 12054b167c7a649a7aaacf6e663f5fac parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0212840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 34 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855820/e4f80ceb-3483-453a-aac6-2222e463e2a6_characterizeImage_855820_2.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855820/e4f80ceb-3483-453a-aac6-2222e463e2a6_characterizeImage_855820_2.stderr parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 53 submitted for App calibrate, waiting on task 52 parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 53 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 54 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 54 has memoization hash 194c1e7d0b4f56cf4fa0a12795269d4e parsl.dataflow.memoization INFO: Task 54 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0213ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 35 parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/c73aa34e-5b41-499a-b16f-657e63b319c2_characterizeImage_855910_1.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/c73aa34e-5b41-499a-b16f-657e63b319c2_characterizeImage_855910_1.stderr parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 55 submitted for App calibrate, waiting on task 54 parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 55 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 56 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 56 has memoization hash 2d4744912aceb1810e9243c1b43cc8d4 parsl.dataflow.memoization INFO: Task 56 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec200e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 36 parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/66bb60c0-5531-4492-b309-e5bd6388fe1d_characterizeImage_855911_28.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/66bb60c0-5531-4492-b309-e5bd6388fe1d_characterizeImage_855911_28.stderr parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 57 submitted for App calibrate, waiting on task 56 parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 57 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 58 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 58 has memoization hash 4ca04ffbdd3ef0e35306378e0ba39cfb parsl.dataflow.memoization INFO: Task 58 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec20180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 37 parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/5cabd763-dbeb-4e4d-a653-456299812857_characterizeImage_855911_18.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/5cabd763-dbeb-4e4d-a653-456299812857_characterizeImage_855911_18.stderr parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 59 submitted for App calibrate, waiting on task 58 parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 59 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 60 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 60 has memoization hash 841e65085f790d02600b13fc682951a5 parsl.dataflow.memoization INFO: Task 60 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec202c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 38 parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/f090aa20-a60f-4eb2-a122-cac87a55d70b_characterizeImage_855910_14.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/f090aa20-a60f-4eb2-a122-cac87a55d70b_characterizeImage_855910_14.stderr parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 61 submitted for App calibrate, waiting on task 60 parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 61 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 62 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 62 has memoization hash 5df1050a7ee8ebc372b04620571f0fc6 parsl.dataflow.memoization INFO: Task 62 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec204a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 39 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/529cfa1c-af53-4ac8-99ff-24ac46e84a8c_characterizeImage_855910_26.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/529cfa1c-af53-4ac8-99ff-24ac46e84a8c_characterizeImage_855910_26.stderr parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 63 submitted for App calibrate, waiting on task 62 parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 63 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 64 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash a3b8a455c12f326ea503d6fdfc377777 parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec20680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 40 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/5cab9c4c-303d-4525-8e58-ae2196b669a9_characterizeImage_855911_12.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/5cab9c4c-303d-4525-8e58-ae2196b669a9_characterizeImage_855911_12.stderr parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 65 submitted for App calibrate, waiting on task 64 parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 65 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 66 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 66 has memoization hash 6b1738bbc37c2be15f7810fb8d8af598 parsl.dataflow.memoization INFO: Task 66 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec20a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 41 parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855822/ea0e240a-012a-4c18-88ba-5ff6db71a5ef_characterizeImage_855822_2.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855822/ea0e240a-012a-4c18-88ba-5ff6db71a5ef_characterizeImage_855822_2.stderr parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 67 submitted for App calibrate, waiting on task 66 parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 67 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 68 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 68 has memoization hash 916617b07a87a0ca3525852cab0abe5b parsl.dataflow.memoization INFO: Task 68 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec20860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 42 parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/1ec4b3f8-4b13-4fee-b48a-47c0fa0a0780_characterizeImage_855910_49.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/1ec4b3f8-4b13-4fee-b48a-47c0fa0a0780_characterizeImage_855910_49.stderr parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 69 submitted for App calibrate, waiting on task 68 parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 69 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 70 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash 33e26fbdc1fbf2088ecc6cf991dd7652 parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec20c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 43 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/1c3713cf-9f09-41c4-a287-2ef9026262da_characterizeImage_855910_58.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/1c3713cf-9f09-41c4-a287-2ef9026262da_characterizeImage_855910_58.stderr parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 71 submitted for App calibrate, waiting on task 70 parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 71 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 72 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 72 has memoization hash 7aae0460c0c17a98effb8d617b8de6f8 parsl.dataflow.memoization INFO: Task 72 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec20e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 44 parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/604d91d5-c52a-4c29-a960-33f2108dbba4_characterizeImage_855911_6.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/604d91d5-c52a-4c29-a960-33f2108dbba4_characterizeImage_855911_6.stderr parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 73 submitted for App calibrate, waiting on task 72 parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 73 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 74 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 74 has memoization hash b55d017ba070d9dfacdfbfe16b04c232 parsl.dataflow.memoization INFO: Task 74 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec20fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 45 parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/1ec0a70e-4d13-451d-93b7-bb739eb96816_characterizeImage_855911_45.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/1ec0a70e-4d13-451d-93b7-bb739eb96816_characterizeImage_855911_45.stderr parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 75 submitted for App calibrate, waiting on task 74 parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 75 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 76 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash 1f8a55e3adbb84f41a38bdb1ec792207 parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec213a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 46 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/c39b5901-952f-4dbb-8429-4784cd1d4b0b_characterizeImage_855911_44.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/c39b5901-952f-4dbb-8429-4784cd1d4b0b_characterizeImage_855911_44.stderr parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 77 submitted for App calibrate, waiting on task 76 parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 77 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 78 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 78 has memoization hash 20dbd507b90cddb48c5bec60f75cd595 parsl.dataflow.memoization INFO: Task 78 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec211c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 47 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/1d3d4c4e-0c48-4edb-8599-e62a2b871238_characterizeImage_855910_61.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/1d3d4c4e-0c48-4edb-8599-e62a2b871238_characterizeImage_855910_61.stderr parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 79 submitted for App calibrate, waiting on task 78 parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 79 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 80 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 80 has memoization hash cec517821c7f0321270b94ac5c05e9ff parsl.dataflow.memoization INFO: Task 80 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec21580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 48 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/8e9e637d-c542-4349-9402-9cb673ae28b0_characterizeImage_855910_32.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/8e9e637d-c542-4349-9402-9cb673ae28b0_characterizeImage_855910_32.stderr parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 81 submitted for App calibrate, waiting on task 80 parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 81 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 82 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash 56412489eeec511593cf20792cabf282 parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec21760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 49 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/905952c7-5ff6-4620-bf10-5a34eb50bbfe_characterizeImage_855910_28.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/905952c7-5ff6-4620-bf10-5a34eb50bbfe_characterizeImage_855910_28.stderr parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 83 submitted for App calibrate, waiting on task 82 parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 83 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 84 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 84 has memoization hash 7db659a4949028ff9d17569a29b879b0 parsl.dataflow.memoization INFO: Task 84 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec21940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 50 parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/5a8f3d07-cc65-42a3-8325-39d418efaa04_characterizeImage_855911_29.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/5a8f3d07-cc65-42a3-8325-39d418efaa04_characterizeImage_855911_29.stderr parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 85 submitted for App calibrate, waiting on task 84 parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 85 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 86 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 86 has memoization hash ecc45e6170d2818c2679a3b16a875ceb parsl.dataflow.memoization INFO: Task 86 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec21d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 51 parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/ba5b3c5b-c69c-4f23-a638-8d109708f1f9_characterizeImage_855911_40.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/ba5b3c5b-c69c-4f23-a638-8d109708f1f9_characterizeImage_855911_40.stderr parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 87 submitted for App calibrate, waiting on task 86 parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 87 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 88 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 88 has memoization hash 9a8a62e864cf0fd8f5f3e85efdb9fd59 parsl.dataflow.memoization INFO: Task 88 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec21b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 52 parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/482b7420-a0fa-49f9-9de2-7fb3d7a1456b_characterizeImage_855910_5.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/482b7420-a0fa-49f9-9de2-7fb3d7a1456b_characterizeImage_855910_5.stderr parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 89 submitted for App calibrate, waiting on task 88 parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 89 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 90 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 90 has memoization hash c9f7122a99cb5f462baba291a37979ca parsl.dataflow.memoization INFO: Task 90 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec21ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 53 parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/c88e7d7f-9e9f-494d-9118-e4e91bb318b3_characterizeImage_855910_45.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/c88e7d7f-9e9f-494d-9118-e4e91bb318b3_characterizeImage_855910_45.stderr parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 91 submitted for App calibrate, waiting on task 90 parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 91 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 92 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash e668e5029f3cc4355d61e39ce062ae33 parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec220c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 54 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/336be782-3ef6-4c4e-8fff-25113c3002d8_characterizeImage_855910_40.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/336be782-3ef6-4c4e-8fff-25113c3002d8_characterizeImage_855910_40.stderr parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 93 submitted for App calibrate, waiting on task 92 parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 93 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 94 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 94 has memoization hash d95271db41ba59854c87f2cbba13d89f parsl.dataflow.memoization INFO: Task 94 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec222a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 55 parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/50ff64ee-d8fa-4799-b889-922042a7be83_characterizeImage_855911_23.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/50ff64ee-d8fa-4799-b889-922042a7be83_characterizeImage_855911_23.stderr parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 95 submitted for App calibrate, waiting on task 94 parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 95 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 96 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 96 has memoization hash 31effd051c9b026c54abed0e44d9d24f parsl.dataflow.memoization INFO: Task 96 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec22660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 56 parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/880f73cb-49c4-4049-b1b9-ca8ab2091500_characterizeImage_855911_2.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/880f73cb-49c4-4049-b1b9-ca8ab2091500_characterizeImage_855911_2.stderr parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 97 submitted for App calibrate, waiting on task 96 parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 97 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 98 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 98 has memoization hash debbfeca5e01ca676be2aa833fe92b16 parsl.dataflow.memoization INFO: Task 98 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec22480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 57 parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/fcf9a600-65ab-454e-9347-efced3f152be_characterizeImage_855910_15.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/fcf9a600-65ab-454e-9347-efced3f152be_characterizeImage_855910_15.stderr parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 99 submitted for App calibrate, waiting on task 98 parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 99 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 100 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash 12f48c0cf5a2887697ac9554a734c4fe parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec22840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 58 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/6766e172-c6fd-4873-9723-cd77771eb1bd_characterizeImage_855910_8.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/6766e172-c6fd-4873-9723-cd77771eb1bd_characterizeImage_855910_8.stderr parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 101 submitted for App calibrate, waiting on task 100 parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 101 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 102 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 102 has memoization hash ab3806fbb694140aa958f8e3ef225979 parsl.dataflow.memoization INFO: Task 102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec22a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 59 parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/bd8244e3-e062-4ec0-a4ed-f59d0d413940_characterizeImage_855910_19.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/bd8244e3-e062-4ec0-a4ed-f59d0d413940_characterizeImage_855910_19.stderr parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 103 submitted for App calibrate, waiting on task 102 parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 103 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 104 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 104 has memoization hash 04c191a1fe343b963abedb882721781f parsl.dataflow.memoization INFO: Task 104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec22c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 60 parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/98e11768-2250-42f8-90b4-f807d804f98c_characterizeImage_855910_25.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/98e11768-2250-42f8-90b4-f807d804f98c_characterizeImage_855910_25.stderr parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 105 submitted for App calibrate, waiting on task 104 parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 105 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 106 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash 3214a44fb0bc6e3a79673fbcc3396199 parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec22fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 61 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/52cac3f5-dbc4-475d-b6b6-bae04ea6ee76_characterizeImage_855911_31.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/52cac3f5-dbc4-475d-b6b6-bae04ea6ee76_characterizeImage_855911_31.stderr parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 107 submitted for App calibrate, waiting on task 106 parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 107 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 108 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 108 has memoization hash 9d8f97b8e1cde209a2efeffc8ae6480a parsl.dataflow.memoization INFO: Task 108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec22de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 62 parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/ab699881-3e42-4cd7-aaea-c25fb4618495_characterizeImage_855911_17.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/ab699881-3e42-4cd7-aaea-c25fb4618495_characterizeImage_855911_17.stderr parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 109 submitted for App calibrate, waiting on task 108 parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 109 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 110 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 110 has memoization hash 40ae38374454a53ca561820a6f98841b parsl.dataflow.memoization INFO: Task 110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec231a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 63 parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/998f5281-04bd-4d22-8886-5c8fb108044a_characterizeImage_855910_2.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/998f5281-04bd-4d22-8886-5c8fb108044a_characterizeImage_855910_2.stderr parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 111 submitted for App calibrate, waiting on task 110 parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 111 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 112 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 112 has memoization hash e2666f3d8b9748f98ae027abbca45960 parsl.dataflow.memoization INFO: Task 112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec23380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 64 parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/98714589-92e8-4855-973f-8251c4b5f8c9_characterizeImage_855911_61.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/98714589-92e8-4855-973f-8251c4b5f8c9_characterizeImage_855911_61.stderr parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 113 submitted for App calibrate, waiting on task 112 parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 113 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 114 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 114 has memoization hash c27671771e741ba6ba212498da98a89a parsl.dataflow.memoization INFO: Task 114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec23560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 65 parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/5ac1ce51-aedf-4f82-9db8-8ec41a8abf75_characterizeImage_855911_4.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/5ac1ce51-aedf-4f82-9db8-8ec41a8abf75_characterizeImage_855911_4.stderr parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 115 submitted for App calibrate, waiting on task 114 parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 115 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 116 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 116 has memoization hash b8841e91316276e5dd6891d38acb698c parsl.dataflow.memoization INFO: Task 116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec23920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 66 parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/ba6bd293-3d10-4464-ba8d-d203f2b41a51_characterizeImage_855911_46.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/ba6bd293-3d10-4464-ba8d-d203f2b41a51_characterizeImage_855911_46.stderr parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 117 submitted for App calibrate, waiting on task 116 parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 117 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 118 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash 6d57f783396e1d260a5a5ee9cd4bdb2c parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec23740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 67 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/3f32effb-4971-4cd6-a6fb-07c39cefa4e0_characterizeImage_855910_33.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/3f32effb-4971-4cd6-a6fb-07c39cefa4e0_characterizeImage_855910_33.stderr parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 119 submitted for App calibrate, waiting on task 118 parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 119 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 120 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 120 has memoization hash 7e22f0114c0ed233b4b693665ac93125 parsl.dataflow.memoization INFO: Task 120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec23b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 68 parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/557817d1-e506-4563-a201-83fb89cacc81_characterizeImage_855910_20.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/557817d1-e506-4563-a201-83fb89cacc81_characterizeImage_855910_20.stderr parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 121 submitted for App calibrate, waiting on task 120 parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 121 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 122 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 122 has memoization hash f08c8251214a6eb92bf951472fb4501e parsl.dataflow.memoization INFO: Task 122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec23ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 69 parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/b24378e6-0128-4e39-8954-9690a15457bd_characterizeImage_855911_47.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/b24378e6-0128-4e39-8954-9690a15457bd_characterizeImage_855911_47.stderr parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 123 submitted for App calibrate, waiting on task 122 parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 123 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 124 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 124 has memoization hash 783324ae5e67dbaeabe63595c1dd6d5e parsl.dataflow.memoization INFO: Task 124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec74040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 70 parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/76337c37-3ebd-49be-b1db-586b608487b7_characterizeImage_855911_20.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/76337c37-3ebd-49be-b1db-586b608487b7_characterizeImage_855911_20.stderr parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 125 submitted for App calibrate, waiting on task 124 parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 125 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 126 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 126 has memoization hash 6b781062ea895e02e9694ec3fd22d6dd parsl.dataflow.memoization INFO: Task 126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec742c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 71 parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/75228b8e-af24-4ef7-a2ab-6ad77397b1f6_characterizeImage_855910_39.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/75228b8e-af24-4ef7-a2ab-6ad77397b1f6_characterizeImage_855910_39.stderr parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 127 submitted for App calibrate, waiting on task 126 parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 127 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 128 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 128 has memoization hash 547525b8c5361f46105c6721959871a8 parsl.dataflow.memoization INFO: Task 128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec740e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 72 parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/b9507bfd-4907-420b-89e8-49e3d290e347_characterizeImage_855910_13.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/b9507bfd-4907-420b-89e8-49e3d290e347_characterizeImage_855910_13.stderr parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 129 submitted for App calibrate, waiting on task 128 parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 129 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 130 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 130 has memoization hash 4f1947c73e07e6a03abda598178896eb parsl.dataflow.memoization INFO: Task 130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec744a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 73 parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/530393c1-16cf-4683-9451-8f61161fb9ab_characterizeImage_855911_33.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/530393c1-16cf-4683-9451-8f61161fb9ab_characterizeImage_855911_33.stderr parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 131 submitted for App calibrate, waiting on task 130 parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 131 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 132 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 132 has memoization hash b2f76cb283beaf8b0d7e287f6c52b1f4 parsl.dataflow.memoization INFO: Task 132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec74680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 74 parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855821/4c2d509c-480b-4b38-8207-7abd607c2208_characterizeImage_855821_2.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855821/4c2d509c-480b-4b38-8207-7abd607c2208_characterizeImage_855821_2.stderr parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 133 submitted for App calibrate, waiting on task 132 parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 133 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 134 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 134 has memoization hash f4b087fca69cbd10e0ea212117e8ab18 parsl.dataflow.memoization INFO: Task 134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec74860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 75 parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/1ac43a9a-71de-4903-bf82-a267193afbc2_characterizeImage_855911_53.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/1ac43a9a-71de-4903-bf82-a267193afbc2_characterizeImage_855911_53.stderr parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 135 submitted for App calibrate, waiting on task 134 parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 135 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 136 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 136 has memoization hash 85c07d4797533be7aacba981040dfd08 parsl.dataflow.memoization INFO: Task 136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec74c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 76 parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/0c0696ad-bcb9-4ee0-bd31-858ea5955eee_characterizeImage_855911_19.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/0c0696ad-bcb9-4ee0-bd31-858ea5955eee_characterizeImage_855911_19.stderr parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 137 submitted for App calibrate, waiting on task 136 parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 137 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 138 submitted for App characterizeImage, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 138 has memoization hash 8c00d28bc925c30b332208303fa03956 parsl.dataflow.memoization INFO: Task 138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec74a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 77 parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/7ed6d274-7055-4b37-9f85-83443fae8a1f_characterizeImage_855911_8.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/7ed6d274-7055-4b37-9f85-83443fae8a1f_characterizeImage_855911_8.stderr parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 139 submitted for App calibrate, waiting on task 138 parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 139 has outstanding dependencies, so launch_if_ready skipping 2024-03-13 01:49:39 proc_lsst.multi:146 [INFO] found job 23564 in provider local proc_lsst.multi INFO: found job 23564 in provider local parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 77 parsl.dataflow.strategy DEBUG: Executor multi has 77 active tasks, 1/0 running/pending blocks, and 0 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 19 more blocks parsl.executors.status_handling INFO: Scaling out by 19 blocks parsl.executors.status_handling INFO: Allocated block ID 1 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1 2024-03-13 01:49:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 2024-03-13 01:49:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 01:49:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 01:49:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 01:49:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 01:49:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-13 01:49:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 01:49:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 01:49:39 proc_lsst.multi:170 [INFO] [multi] job_id 17006978 proc_lsst.multi INFO: [multi] job_id 17006978 2024-03-13 01:49:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 01:49:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 01:49:39 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006978 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006978 parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 17006978 parsl.executors.status_handling INFO: Allocated block ID 2 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2 2024-03-13 01:49:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 2024-03-13 01:49:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 01:49:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 01:49:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 01:49:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 01:49:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 01:49:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 01:49:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 01:49:39 proc_lsst.multi:170 [INFO] [multi] job_id 17006979 proc_lsst.multi INFO: [multi] job_id 17006979 2024-03-13 01:49:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-13 01:49:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 01:49:39 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006979 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006979 parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 17006979 parsl.executors.status_handling INFO: Allocated block ID 3 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3 2024-03-13 01:49:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 2024-03-13 01:49:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 01:49:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 01:49:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 01:49:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 01:49:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-13 01:49:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 01:49:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 01:49:40 proc_lsst.multi:170 [INFO] [multi] job_id 17006980 proc_lsst.multi INFO: [multi] job_id 17006980 2024-03-13 01:49:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-13 01:49:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 01:49:40 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006980 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006980 parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 17006980 parsl.executors.status_handling INFO: Allocated block ID 4 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4 2024-03-13 01:49:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 2024-03-13 01:49:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 01:49:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 01:49:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 01:49:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 01:49:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-13 01:49:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 01:49:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 01:49:40 proc_lsst.multi:170 [INFO] [multi] job_id 17006981 proc_lsst.multi INFO: [multi] job_id 17006981 2024-03-13 01:49:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-13 01:49:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 01:49:40 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006981 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006981 parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 17006981 parsl.executors.status_handling INFO: Allocated block ID 5 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5 2024-03-13 01:49:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 2024-03-13 01:49:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 01:49:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 01:49:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 01:49:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 01:49:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-13 01:49:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 01:49:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 01:49:40 proc_lsst.multi:170 [INFO] [multi] job_id 17006982 proc_lsst.multi INFO: [multi] job_id 17006982 2024-03-13 01:49:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 01:49:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 01:49:40 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006982 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006982 parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 17006982 parsl.executors.status_handling INFO: Allocated block ID 6 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6 2024-03-13 01:49:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 2024-03-13 01:49:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 01:49:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 01:49:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 01:49:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 01:49:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 01:49:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 01:49:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 01:49:41 proc_lsst.multi:170 [INFO] [multi] job_id 17006983 proc_lsst.multi INFO: [multi] job_id 17006983 2024-03-13 01:49:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-13 01:49:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 01:49:41 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006983 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006983 parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 17006983 parsl.executors.status_handling INFO: Allocated block ID 7 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7 2024-03-13 01:49:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 2024-03-13 01:49:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 01:49:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 01:49:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 01:49:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 01:49:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-13 01:49:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 01:49:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 01:49:41 proc_lsst.multi:170 [INFO] [multi] job_id 17006984 proc_lsst.multi INFO: [multi] job_id 17006984 2024-03-13 01:49:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-13 01:49:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 01:49:41 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006984 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006984 parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 17006984 parsl.executors.status_handling INFO: Allocated block ID 8 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8 2024-03-13 01:49:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 2024-03-13 01:49:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 01:49:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 01:49:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 01:49:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 01:49:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-13 01:49:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 01:49:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 01:49:41 proc_lsst.multi:170 [INFO] [multi] job_id 17006985 proc_lsst.multi INFO: [multi] job_id 17006985 2024-03-13 01:49:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-13 01:49:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 01:49:41 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006985 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006985 parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 17006985 parsl.executors.status_handling INFO: Allocated block ID 9 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9 2024-03-13 01:49:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 2024-03-13 01:49:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 01:49:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 01:49:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 01:49:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 01:49:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-13 01:49:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 01:49:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 01:49:42 proc_lsst.multi:170 [INFO] [multi] job_id 17006986 proc_lsst.multi INFO: [multi] job_id 17006986 2024-03-13 01:49:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-13 01:49:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 01:49:42 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006986 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006986 parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 17006986 parsl.executors.status_handling INFO: Allocated block ID 10 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10 2024-03-13 01:49:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 2024-03-13 01:49:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 01:49:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 01:49:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 01:49:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 01:49:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-13 01:49:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 01:49:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 01:49:42 proc_lsst.multi:170 [INFO] [multi] job_id 17006987 proc_lsst.multi INFO: [multi] job_id 17006987 2024-03-13 01:49:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-13 01:49:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 01:49:42 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006987 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006987 parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 17006987 parsl.executors.status_handling INFO: Allocated block ID 11 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11 2024-03-13 01:49:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 2024-03-13 01:49:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 01:49:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 01:49:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 01:49:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 01:49:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-13 01:49:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 01:49:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 01:49:43 proc_lsst.multi:170 [INFO] [multi] job_id 17006988 proc_lsst.multi INFO: [multi] job_id 17006988 2024-03-13 01:49:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-13 01:49:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 01:49:43 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006988 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006988 parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 17006988 parsl.executors.status_handling INFO: Allocated block ID 12 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12 2024-03-13 01:49:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 2024-03-13 01:49:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 01:49:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 01:49:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 01:49:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 01:49:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-13 01:49:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 01:49:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 01:49:43 proc_lsst.multi:170 [INFO] [multi] job_id 17006989 proc_lsst.multi INFO: [multi] job_id 17006989 2024-03-13 01:49:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-13 01:49:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 01:49:43 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006989 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006989 parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 17006989 parsl.executors.status_handling INFO: Allocated block ID 13 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13 2024-03-13 01:49:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 2024-03-13 01:49:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 01:49:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 01:49:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 01:49:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 01:49:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-13 01:49:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 01:49:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 01:49:44 proc_lsst.multi:170 [INFO] [multi] job_id 17006990 proc_lsst.multi INFO: [multi] job_id 17006990 2024-03-13 01:49:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-13 01:49:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 01:49:44 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006990 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006990 parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 17006990 parsl.executors.status_handling INFO: Allocated block ID 14 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14 2024-03-13 01:49:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 2024-03-13 01:49:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 01:49:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 01:49:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 01:49:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 01:49:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-13 01:49:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 01:49:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 01:49:44 proc_lsst.multi:170 [INFO] [multi] job_id 17006991 proc_lsst.multi INFO: [multi] job_id 17006991 2024-03-13 01:49:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-13 01:49:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 01:49:44 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006991 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006991 parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 17006991 parsl.executors.status_handling INFO: Allocated block ID 15 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15 2024-03-13 01:49:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 2024-03-13 01:49:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 01:49:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 01:49:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 01:49:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 01:49:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-13 01:49:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 01:49:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 01:49:45 proc_lsst.multi:170 [INFO] [multi] job_id 17006992 proc_lsst.multi INFO: [multi] job_id 17006992 2024-03-13 01:49:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-13 01:49:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 01:49:45 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006992 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006992 parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 17006992 parsl.executors.status_handling INFO: Allocated block ID 16 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16 2024-03-13 01:49:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 2024-03-13 01:49:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 01:49:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 01:49:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 01:49:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 01:49:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-13 01:49:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 01:49:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 01:49:45 proc_lsst.multi:170 [INFO] [multi] job_id 17006993 proc_lsst.multi INFO: [multi] job_id 17006993 2024-03-13 01:49:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-13 01:49:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 01:49:45 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006993 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006993 parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 17006993 parsl.executors.status_handling INFO: Allocated block ID 17 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17 2024-03-13 01:49:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 2024-03-13 01:49:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 01:49:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 01:49:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 01:49:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 01:49:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-13 01:49:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 01:49:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 01:49:46 proc_lsst.multi:170 [INFO] [multi] job_id 17006994 proc_lsst.multi INFO: [multi] job_id 17006994 2024-03-13 01:49:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-13 01:49:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 01:49:46 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006994 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006994 parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 17006994 parsl.executors.status_handling INFO: Allocated block ID 18 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18 2024-03-13 01:49:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 2024-03-13 01:49:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 01:49:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 01:49:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 01:49:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 01:49:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-13 01:49:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 01:49:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 01:49:46 proc_lsst.multi:170 [INFO] [multi] job_id 17006995 proc_lsst.multi INFO: [multi] job_id 17006995 2024-03-13 01:49:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-13 01:49:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 01:49:46 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006995 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006995 parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 17006995 parsl.executors.status_handling INFO: Allocated block ID 19 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19 2024-03-13 01:49:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 2024-03-13 01:49:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 01:49:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 01:49:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 01:49:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 01:49:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-13 01:49:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 01:49:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,169.254.95.120,198.48.92.26,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54854 --result_port=54976 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 01:49:47 proc_lsst.multi:170 [INFO] [multi] job_id 17006996 proc_lsst.multi INFO: [multi] job_id 17006996 2024-03-13 01:49:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-13 01:49:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 01:49:47 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17006996 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17006996 parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 17006996 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22667511342224 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 20, active_tasks = 77 parsl.dataflow.strategy DEBUG: Executor multi has 77 active tasks, 1/19 running/pending blocks, and 8 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22667511342224 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 20, active_tasks = 77 parsl.dataflow.strategy DEBUG: Executor multi has 77 active tasks, 1/19 running/pending blocks, and 19 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22667511342224 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 20, active_tasks = 77 parsl.dataflow.strategy DEBUG: Executor multi has 77 active tasks, 1/19 running/pending blocks, and 21 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22667511342224 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 1 try 0 failed parsl.dataflow.dflow INFO: Task 1 marked for retry parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/846708ef-297c-4241-80c4-81276b622400_calibrate_855911_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/846708ef-297c-4241-80c4-81276b622400_calibrate_855911_9.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1 has memoization hash 945eb2730401d3236a5cb33f0270457a parsl.dataflow.memoization INFO: Task 1 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db02109a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1 try 1 launched on executor multi with executor id 78 parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/846708ef-297c-4241-80c4-81276b622400_calibrate_855911_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/846708ef-297c-4241-80c4-81276b622400_calibrate_855911_9.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 20, active_tasks = 77 parsl.dataflow.strategy DEBUG: Executor multi has 77 active tasks, 1/19 running/pending blocks, and 21 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22667511342224 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 3 try 0 failed parsl.dataflow.dflow INFO: Task 3 marked for retry parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/fc6e7088-7176-4422-96f7-861be6218102_calibrate_855911_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/fc6e7088-7176-4422-96f7-861be6218102_calibrate_855911_14.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3 has memoization hash 8060fa9cd95588ec1f90daf399058a28 parsl.dataflow.memoization INFO: Task 3 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0210f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3 try 1 launched on executor multi with executor id 79 parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/fc6e7088-7176-4422-96f7-861be6218102_calibrate_855911_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/fc6e7088-7176-4422-96f7-861be6218102_calibrate_855911_14.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 2 try 0 failed parsl.dataflow.dflow INFO: Task 2 marked for retry parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/1ba9f290-426e-40fd-b776-5140f06d5856_calibrate_855911_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/1ba9f290-426e-40fd-b776-5140f06d5856_calibrate_855911_32.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2 has memoization hash f4bf0c3c4eedcb763ff4ab113b3e09fe parsl.dataflow.memoization INFO: Task 2 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0210ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2 try 1 launched on executor multi with executor id 80 parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/1ba9f290-426e-40fd-b776-5140f06d5856_calibrate_855911_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/1ba9f290-426e-40fd-b776-5140f06d5856_calibrate_855911_32.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 0 try 0 failed parsl.dataflow.dflow INFO: Task 0 marked for retry parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/19a20ea9-58db-4c2b-9e4e-82d1cf7cb8c9_calibrate_855911_35.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/19a20ea9-58db-4c2b-9e4e-82d1cf7cb8c9_calibrate_855911_35.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 0 has memoization hash c877fe17b3afe9eaa5084e2b9088ce24 parsl.dataflow.memoization INFO: Task 0 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0210900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 0 try 1 launched on executor multi with executor id 81 parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/19a20ea9-58db-4c2b-9e4e-82d1cf7cb8c9_calibrate_855911_35.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/19a20ea9-58db-4c2b-9e4e-82d1cf7cb8c9_calibrate_855911_35.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 16 try 0 failed parsl.dataflow.dflow INFO: Task 16 marked for retry parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/9390e644-3687-46b6-aa37-eb7c80d4de29_characterizeImage_855910_16.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/9390e644-3687-46b6-aa37-eb7c80d4de29_characterizeImage_855910_16.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash f919b0be28762e72b432c44f5167d861 parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0211760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 16 try 1 launched on executor multi with executor id 82 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/9390e644-3687-46b6-aa37-eb7c80d4de29_characterizeImage_855910_16.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/9390e644-3687-46b6-aa37-eb7c80d4de29_characterizeImage_855910_16.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 20 try 0 failed parsl.dataflow.dflow INFO: Task 20 marked for retry parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/ca786ff8-5ae7-4c4c-845b-c815ba80c3a2_characterizeImage_855911_15.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/ca786ff8-5ae7-4c4c-845b-c815ba80c3a2_characterizeImage_855911_15.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 20 has memoization hash 7cdaea0c83a308f4fe22ba2e71c4ee16 parsl.dataflow.memoization INFO: Task 20 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0211c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 20 try 1 launched on executor multi with executor id 83 parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/ca786ff8-5ae7-4c4c-845b-c815ba80c3a2_characterizeImage_855911_15.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/ca786ff8-5ae7-4c4c-845b-c815ba80c3a2_characterizeImage_855911_15.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 14 try 0 failed parsl.dataflow.dflow INFO: Task 14 marked for retry parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/08082892-9973-49b4-81e5-e3b332deac5b_characterizeImage_855911_38.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/08082892-9973-49b4-81e5-e3b332deac5b_characterizeImage_855911_38.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 14 has memoization hash 5d5e14c03ea0cf657543b8fa137794fd parsl.dataflow.memoization INFO: Task 14 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0211940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 14 try 1 launched on executor multi with executor id 84 parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/08082892-9973-49b4-81e5-e3b332deac5b_characterizeImage_855911_38.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/08082892-9973-49b4-81e5-e3b332deac5b_characterizeImage_855911_38.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 22 try 0 failed parsl.dataflow.dflow INFO: Task 22 marked for retry parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/800a2922-f721-4d25-822c-a7094ee8caed_characterizeImage_855910_46.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/800a2922-f721-4d25-822c-a7094ee8caed_characterizeImage_855910_46.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash 9f73ec5b9a7aef9a54c21b04299c3355 parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0211f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 1 launched on executor multi with executor id 85 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/800a2922-f721-4d25-822c-a7094ee8caed_characterizeImage_855910_46.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/800a2922-f721-4d25-822c-a7094ee8caed_characterizeImage_855910_46.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 24 try 0 failed parsl.dataflow.dflow INFO: Task 24 marked for retry parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/6175499a-5177-4137-b42d-38fc1c31a263_characterizeImage_855911_58.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/6175499a-5177-4137-b42d-38fc1c31a263_characterizeImage_855911_58.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 24 has memoization hash 843374210022ec35e6ac2113ed5276ce parsl.dataflow.memoization INFO: Task 24 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0212160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 24 try 1 launched on executor multi with executor id 86 parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/6175499a-5177-4137-b42d-38fc1c31a263_characterizeImage_855911_58.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/6175499a-5177-4137-b42d-38fc1c31a263_characterizeImage_855911_58.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 26 try 0 failed parsl.dataflow.dflow INFO: Task 26 marked for retry parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/1a97605c-6432-4995-b290-68cb12b763a2_characterizeImage_855911_5.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/1a97605c-6432-4995-b290-68cb12b763a2_characterizeImage_855911_5.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 26 has memoization hash 1470caf28784b09d64c5bf305806b824 parsl.dataflow.memoization INFO: Task 26 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0212520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 26 try 1 launched on executor multi with executor id 87 parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/1a97605c-6432-4995-b290-68cb12b763a2_characterizeImage_855911_5.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/1a97605c-6432-4995-b290-68cb12b763a2_characterizeImage_855911_5.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 6 try 0 failed parsl.dataflow.dflow INFO: Task 6 marked for retry parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/634fa457-accc-4576-90ce-3d577bc12af2_calibrate_855911_25.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/634fa457-accc-4576-90ce-3d577bc12af2_calibrate_855911_25.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 6 has memoization hash d6db8b96e78af2a4f34530c3c84023ad parsl.dataflow.memoization INFO: Task 6 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0211080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 6 try 1 launched on executor multi with executor id 88 parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/634fa457-accc-4576-90ce-3d577bc12af2_calibrate_855911_25.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/634fa457-accc-4576-90ce-3d577bc12af2_calibrate_855911_25.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 11 try 0 failed parsl.dataflow.dflow INFO: Task 11 marked for retry parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/13eadaed-b6ed-46a9-a1fe-1e9790987a07_calibrate_855910_21.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/13eadaed-b6ed-46a9-a1fe-1e9790987a07_calibrate_855910_21.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 11 has memoization hash 1f0a16be7fea9254b03f305974f9a281 parsl.dataflow.memoization INFO: Task 11 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0211300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 11 try 1 launched on executor multi with executor id 89 parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/13eadaed-b6ed-46a9-a1fe-1e9790987a07_calibrate_855910_21.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/13eadaed-b6ed-46a9-a1fe-1e9790987a07_calibrate_855910_21.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 20, active_tasks = 77 parsl.dataflow.strategy DEBUG: Executor multi has 77 active tasks, 1/19 running/pending blocks, and 21 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22667511342224 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 8 try 0 failed parsl.dataflow.dflow INFO: Task 8 marked for retry parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/63247ced-0399-4767-bf05-e6d64abbfb8d_calibrate_855910_9.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/63247ced-0399-4767-bf05-e6d64abbfb8d_calibrate_855910_9.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 8 has memoization hash 5be10890c0174fc8f64aad9dfabce424 parsl.dataflow.memoization INFO: Task 8 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0210a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 8 try 1 launched on executor multi with executor id 90 parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/63247ced-0399-4767-bf05-e6d64abbfb8d_calibrate_855910_9.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/63247ced-0399-4767-bf05-e6d64abbfb8d_calibrate_855910_9.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 18 try 0 failed parsl.dataflow.dflow INFO: Task 18 marked for retry parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/a090aba4-4a38-4573-a572-f5cabab5b056_characterizeImage_855911_13.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/a090aba4-4a38-4573-a572-f5cabab5b056_characterizeImage_855911_13.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 18 has memoization hash bb34b92b8d6ed10882635aeefc65760a parsl.dataflow.memoization INFO: Task 18 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0211a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 18 try 1 launched on executor multi with executor id 91 parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/a090aba4-4a38-4573-a572-f5cabab5b056_characterizeImage_855911_13.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/a090aba4-4a38-4573-a572-f5cabab5b056_characterizeImage_855911_13.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 9 try 0 failed parsl.dataflow.dflow INFO: Task 9 marked for retry parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/8b2fa882-73b1-4798-923c-386c396bac28_calibrate_855911_27.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/8b2fa882-73b1-4798-923c-386c396bac28_calibrate_855911_27.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 9 has memoization hash 62215dcfe5e4e0955f0641f249b0e10d parsl.dataflow.memoization INFO: Task 9 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0211120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 9 try 1 launched on executor multi with executor id 92 parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/8b2fa882-73b1-4798-923c-386c396bac28_calibrate_855911_27.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/8b2fa882-73b1-4798-923c-386c396bac28_calibrate_855911_27.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 13 try 0 failed parsl.dataflow.dflow INFO: Task 13 marked for retry parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/8784752d-bf08-47df-b8a0-c97ff7f9b4a0_calibrate_855910_4.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/8784752d-bf08-47df-b8a0-c97ff7f9b4a0_calibrate_855910_4.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 13 has memoization hash e4cdb6332ecaf44719ea625b51e0c07e parsl.dataflow.memoization INFO: Task 13 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0211800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 13 try 1 launched on executor multi with executor id 93 parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/8784752d-bf08-47df-b8a0-c97ff7f9b4a0_calibrate_855910_4.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/8784752d-bf08-47df-b8a0-c97ff7f9b4a0_calibrate_855910_4.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 5 try 0 failed parsl.dataflow.dflow INFO: Task 5 marked for retry parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/db2895de-bf84-4a3e-8579-ea13eb770871_calibrate_855911_34.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/db2895de-bf84-4a3e-8579-ea13eb770871_calibrate_855911_34.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5 has memoization hash 7b9f97e5556c134f7b40ed88f2184075 parsl.dataflow.memoization INFO: Task 5 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0210fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5 try 1 launched on executor multi with executor id 94 parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/db2895de-bf84-4a3e-8579-ea13eb770871_calibrate_855911_34.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/db2895de-bf84-4a3e-8579-ea13eb770871_calibrate_855911_34.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 10 try 0 failed parsl.dataflow.dflow INFO: Task 10 marked for retry parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/89d5f5c0-6cde-42f9-9981-aa0a6946ff66_calibrate_855910_34.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/89d5f5c0-6cde-42f9-9981-aa0a6946ff66_calibrate_855910_34.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash 569c5fab366f4d0989df9b03d78cace5 parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0210e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 10 try 1 launched on executor multi with executor id 95 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/89d5f5c0-6cde-42f9-9981-aa0a6946ff66_calibrate_855910_34.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/89d5f5c0-6cde-42f9-9981-aa0a6946ff66_calibrate_855910_34.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 28 try 0 failed parsl.dataflow.dflow INFO: Task 28 marked for retry parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/983a01f8-c7bf-4fef-84c8-a8caf58c74b6_characterizeImage_855911_39.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/983a01f8-c7bf-4fef-84c8-a8caf58c74b6_characterizeImage_855911_39.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash 8aafad68ee30a513d7ff0fc7ce1f5c26 parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db02120c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 1 launched on executor multi with executor id 96 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/983a01f8-c7bf-4fef-84c8-a8caf58c74b6_characterizeImage_855911_39.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/983a01f8-c7bf-4fef-84c8-a8caf58c74b6_characterizeImage_855911_39.stderr 2024-03-13 01:50:12 proc_lsst.multi:146 [INFO] found job 23564 in provider local proc_lsst.multi INFO: found job 23564 in provider local 2024-03-13 01:50:12 proc_lsst.multi:146 [INFO] found job 17006978 in provider astro proc_lsst.multi INFO: found job 17006978 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:50:12 proc_lsst.multi:146 [INFO] found job 17006979 in provider astro proc_lsst.multi INFO: found job 17006979 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:50:13 proc_lsst.multi:146 [INFO] found job 17006980 in provider astro proc_lsst.multi INFO: found job 17006980 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:50:13 proc_lsst.multi:146 [INFO] found job 17006981 in provider astro proc_lsst.multi INFO: found job 17006981 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 48 try 0 failed parsl.dataflow.dflow INFO: Task 48 marked for retry parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855909/d4097b91-b153-43b0-866d-30e0c77604e5_characterizeImage_855909_2.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855909/d4097b91-b153-43b0-866d-30e0c77604e5_characterizeImage_855909_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 48 has memoization hash 7ee445c26b0e81f76caf54fd1e2daa1c parsl.dataflow.memoization INFO: Task 48 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0212de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 48 try 1 launched on executor multi with executor id 97 parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855909/d4097b91-b153-43b0-866d-30e0c77604e5_characterizeImage_855909_2.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855909/d4097b91-b153-43b0-866d-30e0c77604e5_characterizeImage_855909_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:50:14 proc_lsst.multi:146 [INFO] found job 17006982 in provider astro proc_lsst.multi INFO: found job 17006982 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:50:14 proc_lsst.multi:146 [INFO] found job 17006983 in provider astro proc_lsst.multi INFO: found job 17006983 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 4 try 0 failed parsl.dataflow.dflow INFO: Task 4 marked for retry parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/64c54979-dfdb-429a-897d-cbd231b11b96_calibrate_855911_59.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/64c54979-dfdb-429a-897d-cbd231b11b96_calibrate_855911_59.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash f5dc0abcdf5fdb8a5a6b8ffcfc7941a0 parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0210d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4 try 1 launched on executor multi with executor id 98 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/64c54979-dfdb-429a-897d-cbd231b11b96_calibrate_855911_59.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/64c54979-dfdb-429a-897d-cbd231b11b96_calibrate_855911_59.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 52 try 0 failed parsl.dataflow.dflow INFO: Task 52 marked for retry parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855820/e4f80ceb-3483-453a-aac6-2222e463e2a6_characterizeImage_855820_2.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855820/e4f80ceb-3483-453a-aac6-2222e463e2a6_characterizeImage_855820_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash 12054b167c7a649a7aaacf6e663f5fac parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0212840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 1 launched on executor multi with executor id 99 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855820/e4f80ceb-3483-453a-aac6-2222e463e2a6_characterizeImage_855820_2.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855820/e4f80ceb-3483-453a-aac6-2222e463e2a6_characterizeImage_855820_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:50:15 proc_lsst.multi:146 [INFO] found job 17006984 in provider astro proc_lsst.multi INFO: found job 17006984 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:50:15 proc_lsst.multi:146 [INFO] found job 17006985 in provider astro proc_lsst.multi INFO: found job 17006985 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 30 try 0 failed parsl.dataflow.dflow INFO: Task 30 marked for retry parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/5c0d4d51-2440-430b-9231-d536d662efe7_characterizeImage_855911_11.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/5c0d4d51-2440-430b-9231-d536d662efe7_characterizeImage_855911_11.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 30 has memoization hash c8f9f8853974013de48d830f921f325c parsl.dataflow.memoization INFO: Task 30 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0212660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 30 try 1 launched on executor multi with executor id 100 parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/5c0d4d51-2440-430b-9231-d536d662efe7_characterizeImage_855911_11.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/5c0d4d51-2440-430b-9231-d536d662efe7_characterizeImage_855911_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:50:16 proc_lsst.multi:146 [INFO] found job 17006986 in provider astro proc_lsst.multi INFO: found job 17006986 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 42 try 0 failed parsl.dataflow.dflow INFO: Task 42 marked for retry parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/8774d934-8e13-4ee7-9af4-b8633afb8117_characterizeImage_855911_3.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/8774d934-8e13-4ee7-9af4-b8633afb8117_characterizeImage_855911_3.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 42 has memoization hash dd11e2e4560735a94dba9365e9b48f75 parsl.dataflow.memoization INFO: Task 42 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db02131a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 42 try 1 launched on executor multi with executor id 101 parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/8774d934-8e13-4ee7-9af4-b8633afb8117_characterizeImage_855911_3.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/8774d934-8e13-4ee7-9af4-b8633afb8117_characterizeImage_855911_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:50:17 proc_lsst.multi:146 [INFO] found job 17006987 in provider astro proc_lsst.multi INFO: found job 17006987 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 34 try 0 failed parsl.dataflow.dflow INFO: Task 34 marked for retry parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/4f91a3bb-dde6-45ad-ae80-2cec03be0edf_characterizeImage_855911_16.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/4f91a3bb-dde6-45ad-ae80-2cec03be0edf_characterizeImage_855911_16.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash 075cfe43ceab10b0e0991f5e1ce3038b parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0213920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 1 launched on executor multi with executor id 102 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/4f91a3bb-dde6-45ad-ae80-2cec03be0edf_characterizeImage_855911_16.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/4f91a3bb-dde6-45ad-ae80-2cec03be0edf_characterizeImage_855911_16.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 38 try 0 failed parsl.dataflow.dflow INFO: Task 38 marked for retry parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/182ff692-3216-4bc9-a828-1736d1daf016_characterizeImage_855911_1.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/182ff692-3216-4bc9-a828-1736d1daf016_characterizeImage_855911_1.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 38 has memoization hash 279fc759a984b5b6d09ebe68191901a9 parsl.dataflow.memoization INFO: Task 38 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0213740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 38 try 1 launched on executor multi with executor id 103 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/182ff692-3216-4bc9-a828-1736d1daf016_characterizeImage_855911_1.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/182ff692-3216-4bc9-a828-1736d1daf016_characterizeImage_855911_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:50:17 proc_lsst.multi:146 [INFO] found job 17006988 in provider astro proc_lsst.multi INFO: found job 17006988 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 40 try 0 failed parsl.dataflow.dflow INFO: Task 40 marked for retry parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/c82fb047-b7dc-49ad-845d-7d9c7cc1ecd1_characterizeImage_855910_47.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/c82fb047-b7dc-49ad-845d-7d9c7cc1ecd1_characterizeImage_855910_47.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash 695315fd2af3281397c8bf1391395ac0 parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0213380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 40 try 1 launched on executor multi with executor id 104 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/c82fb047-b7dc-49ad-845d-7d9c7cc1ecd1_characterizeImage_855910_47.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/c82fb047-b7dc-49ad-845d-7d9c7cc1ecd1_characterizeImage_855910_47.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 36 try 0 failed parsl.dataflow.dflow INFO: Task 36 marked for retry parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/256f3f22-a5e3-4dc6-bfbf-67a9cd648029_characterizeImage_855911_21.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/256f3f22-a5e3-4dc6-bfbf-67a9cd648029_characterizeImage_855911_21.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 36 has memoization hash 3e33fcbdba0efe15fcc1181bee9baaaf parsl.dataflow.memoization INFO: Task 36 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0213560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 36 try 1 launched on executor multi with executor id 105 parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/256f3f22-a5e3-4dc6-bfbf-67a9cd648029_characterizeImage_855911_21.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/256f3f22-a5e3-4dc6-bfbf-67a9cd648029_characterizeImage_855911_21.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 32 try 0 failed parsl.dataflow.dflow INFO: Task 32 marked for retry parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/8d6c5ccf-e869-40b5-8751-c8d8d42c408d_characterizeImage_855910_27.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/8d6c5ccf-e869-40b5-8751-c8d8d42c408d_characterizeImage_855910_27.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 32 has memoization hash a38cb225b314a6deb4f26a38c497b849 parsl.dataflow.memoization INFO: Task 32 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0213a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 32 try 1 launched on executor multi with executor id 106 parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/8d6c5ccf-e869-40b5-8751-c8d8d42c408d_characterizeImage_855910_27.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/8d6c5ccf-e869-40b5-8751-c8d8d42c408d_characterizeImage_855910_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:50:18 proc_lsst.multi:146 [INFO] found job 17006989 in provider astro proc_lsst.multi INFO: found job 17006989 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 44 try 0 failed parsl.dataflow.dflow INFO: Task 44 marked for retry parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/22235225-dfff-445b-b762-eec54521fba4_characterizeImage_855910_51.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/22235225-dfff-445b-b762-eec54521fba4_characterizeImage_855910_51.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 44 has memoization hash 3bff9b078737f4a7145ba0c4f60a220c parsl.dataflow.memoization INFO: Task 44 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0212fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 44 try 1 launched on executor multi with executor id 107 parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/22235225-dfff-445b-b762-eec54521fba4_characterizeImage_855910_51.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/22235225-dfff-445b-b762-eec54521fba4_characterizeImage_855910_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:50:18 proc_lsst.multi:146 [INFO] found job 17006990 in provider astro proc_lsst.multi INFO: found job 17006990 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 50 try 0 failed parsl.dataflow.dflow INFO: Task 50 marked for retry parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/0a7de8f6-9d73-4bcc-9593-33726fcaba42_characterizeImage_855911_48.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/0a7de8f6-9d73-4bcc-9593-33726fcaba42_characterizeImage_855911_48.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 50 has memoization hash c81f729b57ed59987b5c33c48e4370ad parsl.dataflow.memoization INFO: Task 50 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0212a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 50 try 1 launched on executor multi with executor id 108 parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/0a7de8f6-9d73-4bcc-9593-33726fcaba42_characterizeImage_855911_48.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/0a7de8f6-9d73-4bcc-9593-33726fcaba42_characterizeImage_855911_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:50:19 proc_lsst.multi:146 [INFO] found job 17006991 in provider astro proc_lsst.multi INFO: found job 17006991 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:50:19 proc_lsst.multi:146 [INFO] found job 17006992 in provider astro proc_lsst.multi INFO: found job 17006992 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:50:19 proc_lsst.multi:146 [INFO] found job 17006993 in provider astro proc_lsst.multi INFO: found job 17006993 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 56 try 0 failed parsl.dataflow.dflow INFO: Task 56 marked for retry parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/66bb60c0-5531-4492-b309-e5bd6388fe1d_characterizeImage_855911_28.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/66bb60c0-5531-4492-b309-e5bd6388fe1d_characterizeImage_855911_28.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 56 has memoization hash 2d4744912aceb1810e9243c1b43cc8d4 parsl.dataflow.memoization INFO: Task 56 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec200e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 56 try 1 launched on executor multi with executor id 109 parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/66bb60c0-5531-4492-b309-e5bd6388fe1d_characterizeImage_855911_28.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/66bb60c0-5531-4492-b309-e5bd6388fe1d_characterizeImage_855911_28.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 46 try 0 failed parsl.dataflow.dflow INFO: Task 46 marked for retry parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/640eda07-0934-42c3-a791-049094bd785a_characterizeImage_855911_22.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/640eda07-0934-42c3-a791-049094bd785a_characterizeImage_855911_22.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 46 has memoization hash ef41369a22e73637504962f6c442c39b parsl.dataflow.memoization INFO: Task 46 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0212c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 46 try 1 launched on executor multi with executor id 110 parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/640eda07-0934-42c3-a791-049094bd785a_characterizeImage_855911_22.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/640eda07-0934-42c3-a791-049094bd785a_characterizeImage_855911_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:50:20 proc_lsst.multi:146 [INFO] found job 17006994 in provider astro proc_lsst.multi INFO: found job 17006994 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:50:20 proc_lsst.multi:146 [INFO] found job 17006995 in provider astro proc_lsst.multi INFO: found job 17006995 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 58 try 0 failed parsl.dataflow.dflow INFO: Task 58 marked for retry parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/5cabd763-dbeb-4e4d-a653-456299812857_characterizeImage_855911_18.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/5cabd763-dbeb-4e4d-a653-456299812857_characterizeImage_855911_18.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 58 has memoization hash 4ca04ffbdd3ef0e35306378e0ba39cfb parsl.dataflow.memoization INFO: Task 58 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec20180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 58 try 1 launched on executor multi with executor id 111 parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/5cabd763-dbeb-4e4d-a653-456299812857_characterizeImage_855911_18.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/5cabd763-dbeb-4e4d-a653-456299812857_characterizeImage_855911_18.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 7 try 0 failed parsl.dataflow.dflow INFO: Task 7 marked for retry parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/235a9997-cb8d-4b9d-927a-026f2e0b49dd_calibrate_855911_41.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/235a9997-cb8d-4b9d-927a-026f2e0b49dd_calibrate_855911_41.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 7 has memoization hash a4de438e871dead9fe871ac68864ef3c parsl.dataflow.memoization INFO: Task 7 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0210220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 7 try 1 launched on executor multi with executor id 112 parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/235a9997-cb8d-4b9d-927a-026f2e0b49dd_calibrate_855911_41.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/235a9997-cb8d-4b9d-927a-026f2e0b49dd_calibrate_855911_41.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 54 try 0 failed parsl.dataflow.dflow INFO: Task 54 marked for retry parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/c73aa34e-5b41-499a-b16f-657e63b319c2_characterizeImage_855910_1.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/c73aa34e-5b41-499a-b16f-657e63b319c2_characterizeImage_855910_1.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 54 has memoization hash 194c1e7d0b4f56cf4fa0a12795269d4e parsl.dataflow.memoization INFO: Task 54 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0213ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 54 try 1 launched on executor multi with executor id 113 parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/c73aa34e-5b41-499a-b16f-657e63b319c2_characterizeImage_855910_1.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/c73aa34e-5b41-499a-b16f-657e63b319c2_characterizeImage_855910_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:50:21 proc_lsst.multi:146 [INFO] found job 17006996 in provider astro proc_lsst.multi INFO: found job 17006996 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 66 try 0 failed parsl.dataflow.dflow INFO: Task 66 marked for retry parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855822/ea0e240a-012a-4c18-88ba-5ff6db71a5ef_characterizeImage_855822_2.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855822/ea0e240a-012a-4c18-88ba-5ff6db71a5ef_characterizeImage_855822_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 66 has memoization hash 6b1738bbc37c2be15f7810fb8d8af598 parsl.dataflow.memoization INFO: Task 66 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec20a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 66 try 1 launched on executor multi with executor id 114 parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855822/ea0e240a-012a-4c18-88ba-5ff6db71a5ef_characterizeImage_855822_2.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855822/ea0e240a-012a-4c18-88ba-5ff6db71a5ef_characterizeImage_855822_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 12 try 0 failed parsl.dataflow.dflow INFO: Task 12 marked for retry parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/d136b04a-8eac-4050-8d12-d225ce3c1cf0_calibrate_855910_35.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/d136b04a-8eac-4050-8d12-d225ce3c1cf0_calibrate_855910_35.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 12 has memoization hash b7bcf3fad9715cc4965aac2677681012 parsl.dataflow.memoization INFO: Task 12 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149db0211440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 12 try 1 launched on executor multi with executor id 115 parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/d136b04a-8eac-4050-8d12-d225ce3c1cf0_calibrate_855910_35.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/d136b04a-8eac-4050-8d12-d225ce3c1cf0_calibrate_855910_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 20, active_tasks = 77 parsl.dataflow.strategy DEBUG: Executor multi has 77 active tasks, 20/0 running/pending blocks, and 21 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22667511342224 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 20, active_tasks = 77 parsl.dataflow.strategy DEBUG: Executor multi has 77 active tasks, 20/0 running/pending blocks, and 21 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22667511342224 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 62 try 0 failed parsl.dataflow.dflow INFO: Task 62 marked for retry parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/529cfa1c-af53-4ac8-99ff-24ac46e84a8c_characterizeImage_855910_26.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/529cfa1c-af53-4ac8-99ff-24ac46e84a8c_characterizeImage_855910_26.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 62 has memoization hash 5df1050a7ee8ebc372b04620571f0fc6 parsl.dataflow.memoization INFO: Task 62 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec204a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 62 try 1 launched on executor multi with executor id 116 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/529cfa1c-af53-4ac8-99ff-24ac46e84a8c_characterizeImage_855910_26.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/529cfa1c-af53-4ac8-99ff-24ac46e84a8c_characterizeImage_855910_26.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 60 try 0 failed parsl.dataflow.dflow INFO: Task 60 marked for retry parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/f090aa20-a60f-4eb2-a122-cac87a55d70b_characterizeImage_855910_14.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/f090aa20-a60f-4eb2-a122-cac87a55d70b_characterizeImage_855910_14.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 60 has memoization hash 841e65085f790d02600b13fc682951a5 parsl.dataflow.memoization INFO: Task 60 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec202c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 60 try 1 launched on executor multi with executor id 117 parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/f090aa20-a60f-4eb2-a122-cac87a55d70b_characterizeImage_855910_14.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/f090aa20-a60f-4eb2-a122-cac87a55d70b_characterizeImage_855910_14.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 64 try 0 failed parsl.dataflow.dflow INFO: Task 64 marked for retry parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/5cab9c4c-303d-4525-8e58-ae2196b669a9_characterizeImage_855911_12.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/5cab9c4c-303d-4525-8e58-ae2196b669a9_characterizeImage_855911_12.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash a3b8a455c12f326ea503d6fdfc377777 parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec20680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 1 launched on executor multi with executor id 118 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/5cab9c4c-303d-4525-8e58-ae2196b669a9_characterizeImage_855911_12.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/5cab9c4c-303d-4525-8e58-ae2196b669a9_characterizeImage_855911_12.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 70 try 0 failed parsl.dataflow.dflow INFO: Task 70 marked for retry parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/1c3713cf-9f09-41c4-a287-2ef9026262da_characterizeImage_855910_58.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/1c3713cf-9f09-41c4-a287-2ef9026262da_characterizeImage_855910_58.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash 33e26fbdc1fbf2088ecc6cf991dd7652 parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec20c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 1 launched on executor multi with executor id 119 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/1c3713cf-9f09-41c4-a287-2ef9026262da_characterizeImage_855910_58.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/1c3713cf-9f09-41c4-a287-2ef9026262da_characterizeImage_855910_58.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 20, active_tasks = 77 parsl.dataflow.strategy DEBUG: Executor multi has 77 active tasks, 20/0 running/pending blocks, and 21 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22667511342224 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 72 try 0 failed parsl.dataflow.dflow INFO: Task 72 marked for retry parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/604d91d5-c52a-4c29-a960-33f2108dbba4_characterizeImage_855911_6.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/604d91d5-c52a-4c29-a960-33f2108dbba4_characterizeImage_855911_6.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 72 has memoization hash 7aae0460c0c17a98effb8d617b8de6f8 parsl.dataflow.memoization INFO: Task 72 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec20e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 72 try 1 launched on executor multi with executor id 120 parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/604d91d5-c52a-4c29-a960-33f2108dbba4_characterizeImage_855911_6.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/604d91d5-c52a-4c29-a960-33f2108dbba4_characterizeImage_855911_6.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 74 try 0 failed parsl.dataflow.dflow INFO: Task 74 marked for retry parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/1ec0a70e-4d13-451d-93b7-bb739eb96816_characterizeImage_855911_45.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/1ec0a70e-4d13-451d-93b7-bb739eb96816_characterizeImage_855911_45.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 74 has memoization hash b55d017ba070d9dfacdfbfe16b04c232 parsl.dataflow.memoization INFO: Task 74 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec20fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 74 try 1 launched on executor multi with executor id 121 parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/1ec0a70e-4d13-451d-93b7-bb739eb96816_characterizeImage_855911_45.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/1ec0a70e-4d13-451d-93b7-bb739eb96816_characterizeImage_855911_45.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 96 try 0 failed parsl.dataflow.dflow INFO: Task 96 marked for retry parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/880f73cb-49c4-4049-b1b9-ca8ab2091500_characterizeImage_855911_2.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/880f73cb-49c4-4049-b1b9-ca8ab2091500_characterizeImage_855911_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 96 has memoization hash 31effd051c9b026c54abed0e44d9d24f parsl.dataflow.memoization INFO: Task 96 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec22660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 96 try 1 launched on executor multi with executor id 122 parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/880f73cb-49c4-4049-b1b9-ca8ab2091500_characterizeImage_855911_2.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/880f73cb-49c4-4049-b1b9-ca8ab2091500_characterizeImage_855911_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 68 try 0 failed parsl.dataflow.dflow INFO: Task 68 marked for retry parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/1ec4b3f8-4b13-4fee-b48a-47c0fa0a0780_characterizeImage_855910_49.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/1ec4b3f8-4b13-4fee-b48a-47c0fa0a0780_characterizeImage_855910_49.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 68 has memoization hash 916617b07a87a0ca3525852cab0abe5b parsl.dataflow.memoization INFO: Task 68 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec20860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 68 try 1 launched on executor multi with executor id 123 parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/1ec4b3f8-4b13-4fee-b48a-47c0fa0a0780_characterizeImage_855910_49.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/1ec4b3f8-4b13-4fee-b48a-47c0fa0a0780_characterizeImage_855910_49.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 80 try 0 failed parsl.dataflow.dflow INFO: Task 80 marked for retry parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/8e9e637d-c542-4349-9402-9cb673ae28b0_characterizeImage_855910_32.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/8e9e637d-c542-4349-9402-9cb673ae28b0_characterizeImage_855910_32.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 80 has memoization hash cec517821c7f0321270b94ac5c05e9ff parsl.dataflow.memoization INFO: Task 80 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec21580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 80 try 1 launched on executor multi with executor id 124 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/8e9e637d-c542-4349-9402-9cb673ae28b0_characterizeImage_855910_32.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/8e9e637d-c542-4349-9402-9cb673ae28b0_characterizeImage_855910_32.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 76 try 0 failed parsl.dataflow.dflow INFO: Task 76 marked for retry parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/c39b5901-952f-4dbb-8429-4784cd1d4b0b_characterizeImage_855911_44.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/c39b5901-952f-4dbb-8429-4784cd1d4b0b_characterizeImage_855911_44.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash 1f8a55e3adbb84f41a38bdb1ec792207 parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec213a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 1 launched on executor multi with executor id 125 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/c39b5901-952f-4dbb-8429-4784cd1d4b0b_characterizeImage_855911_44.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/c39b5901-952f-4dbb-8429-4784cd1d4b0b_characterizeImage_855911_44.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 86 try 0 failed parsl.dataflow.dflow INFO: Task 86 marked for retry parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/ba5b3c5b-c69c-4f23-a638-8d109708f1f9_characterizeImage_855911_40.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/ba5b3c5b-c69c-4f23-a638-8d109708f1f9_characterizeImage_855911_40.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 86 has memoization hash ecc45e6170d2818c2679a3b16a875ceb parsl.dataflow.memoization INFO: Task 86 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec21d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 86 try 1 launched on executor multi with executor id 126 parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/ba5b3c5b-c69c-4f23-a638-8d109708f1f9_characterizeImage_855911_40.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/ba5b3c5b-c69c-4f23-a638-8d109708f1f9_characterizeImage_855911_40.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 82 try 0 failed parsl.dataflow.dflow INFO: Task 82 marked for retry parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/905952c7-5ff6-4620-bf10-5a34eb50bbfe_characterizeImage_855910_28.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/905952c7-5ff6-4620-bf10-5a34eb50bbfe_characterizeImage_855910_28.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash 56412489eeec511593cf20792cabf282 parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec21760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 1 launched on executor multi with executor id 127 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/905952c7-5ff6-4620-bf10-5a34eb50bbfe_characterizeImage_855910_28.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/905952c7-5ff6-4620-bf10-5a34eb50bbfe_characterizeImage_855910_28.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 88 try 0 failed parsl.dataflow.dflow INFO: Task 88 marked for retry parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/482b7420-a0fa-49f9-9de2-7fb3d7a1456b_characterizeImage_855910_5.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/482b7420-a0fa-49f9-9de2-7fb3d7a1456b_characterizeImage_855910_5.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 88 has memoization hash 9a8a62e864cf0fd8f5f3e85efdb9fd59 parsl.dataflow.memoization INFO: Task 88 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec21b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 88 try 1 launched on executor multi with executor id 128 parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/482b7420-a0fa-49f9-9de2-7fb3d7a1456b_characterizeImage_855910_5.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/482b7420-a0fa-49f9-9de2-7fb3d7a1456b_characterizeImage_855910_5.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 90 try 0 failed parsl.dataflow.dflow INFO: Task 90 marked for retry parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/c88e7d7f-9e9f-494d-9118-e4e91bb318b3_characterizeImage_855910_45.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/c88e7d7f-9e9f-494d-9118-e4e91bb318b3_characterizeImage_855910_45.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 90 has memoization hash c9f7122a99cb5f462baba291a37979ca parsl.dataflow.memoization INFO: Task 90 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec21ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 90 try 1 launched on executor multi with executor id 129 parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/c88e7d7f-9e9f-494d-9118-e4e91bb318b3_characterizeImage_855910_45.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/c88e7d7f-9e9f-494d-9118-e4e91bb318b3_characterizeImage_855910_45.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 92 try 0 failed parsl.dataflow.dflow INFO: Task 92 marked for retry parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/336be782-3ef6-4c4e-8fff-25113c3002d8_characterizeImage_855910_40.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/336be782-3ef6-4c4e-8fff-25113c3002d8_characterizeImage_855910_40.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash e668e5029f3cc4355d61e39ce062ae33 parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec220c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 1 launched on executor multi with executor id 130 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/336be782-3ef6-4c4e-8fff-25113c3002d8_characterizeImage_855910_40.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/336be782-3ef6-4c4e-8fff-25113c3002d8_characterizeImage_855910_40.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 20, active_tasks = 77 parsl.dataflow.strategy DEBUG: Executor multi has 77 active tasks, 20/0 running/pending blocks, and 21 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22667511342224 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 84 try 0 failed parsl.dataflow.dflow INFO: Task 84 marked for retry parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/5a8f3d07-cc65-42a3-8325-39d418efaa04_characterizeImage_855911_29.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/5a8f3d07-cc65-42a3-8325-39d418efaa04_characterizeImage_855911_29.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 84 has memoization hash 7db659a4949028ff9d17569a29b879b0 parsl.dataflow.memoization INFO: Task 84 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec21940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 84 try 1 launched on executor multi with executor id 131 parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/5a8f3d07-cc65-42a3-8325-39d418efaa04_characterizeImage_855911_29.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/5a8f3d07-cc65-42a3-8325-39d418efaa04_characterizeImage_855911_29.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 110 try 0 failed parsl.dataflow.dflow INFO: Task 110 marked for retry parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/998f5281-04bd-4d22-8886-5c8fb108044a_characterizeImage_855910_2.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/998f5281-04bd-4d22-8886-5c8fb108044a_characterizeImage_855910_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 110 has memoization hash 40ae38374454a53ca561820a6f98841b parsl.dataflow.memoization INFO: Task 110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec231a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 110 try 1 launched on executor multi with executor id 132 parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/998f5281-04bd-4d22-8886-5c8fb108044a_characterizeImage_855910_2.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/998f5281-04bd-4d22-8886-5c8fb108044a_characterizeImage_855910_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 98 try 0 failed parsl.dataflow.dflow INFO: Task 98 marked for retry parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/fcf9a600-65ab-454e-9347-efced3f152be_characterizeImage_855910_15.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/fcf9a600-65ab-454e-9347-efced3f152be_characterizeImage_855910_15.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 98 has memoization hash debbfeca5e01ca676be2aa833fe92b16 parsl.dataflow.memoization INFO: Task 98 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec22480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 98 try 1 launched on executor multi with executor id 133 parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/fcf9a600-65ab-454e-9347-efced3f152be_characterizeImage_855910_15.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/fcf9a600-65ab-454e-9347-efced3f152be_characterizeImage_855910_15.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 102 try 0 failed parsl.dataflow.dflow INFO: Task 102 marked for retry parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/bd8244e3-e062-4ec0-a4ed-f59d0d413940_characterizeImage_855910_19.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/bd8244e3-e062-4ec0-a4ed-f59d0d413940_characterizeImage_855910_19.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 102 has memoization hash ab3806fbb694140aa958f8e3ef225979 parsl.dataflow.memoization INFO: Task 102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec22a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 102 try 1 launched on executor multi with executor id 134 parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/bd8244e3-e062-4ec0-a4ed-f59d0d413940_characterizeImage_855910_19.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/bd8244e3-e062-4ec0-a4ed-f59d0d413940_characterizeImage_855910_19.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 94 try 0 failed parsl.dataflow.dflow INFO: Task 94 marked for retry parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/50ff64ee-d8fa-4799-b889-922042a7be83_characterizeImage_855911_23.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/50ff64ee-d8fa-4799-b889-922042a7be83_characterizeImage_855911_23.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 94 has memoization hash d95271db41ba59854c87f2cbba13d89f parsl.dataflow.memoization INFO: Task 94 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec222a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 94 try 1 launched on executor multi with executor id 135 parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/50ff64ee-d8fa-4799-b889-922042a7be83_characterizeImage_855911_23.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/50ff64ee-d8fa-4799-b889-922042a7be83_characterizeImage_855911_23.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 100 try 0 failed parsl.dataflow.dflow INFO: Task 100 marked for retry parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/6766e172-c6fd-4873-9723-cd77771eb1bd_characterizeImage_855910_8.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/6766e172-c6fd-4873-9723-cd77771eb1bd_characterizeImage_855910_8.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash 12f48c0cf5a2887697ac9554a734c4fe parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec22840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 1 launched on executor multi with executor id 136 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/6766e172-c6fd-4873-9723-cd77771eb1bd_characterizeImage_855910_8.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/6766e172-c6fd-4873-9723-cd77771eb1bd_characterizeImage_855910_8.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 106 try 0 failed parsl.dataflow.dflow INFO: Task 106 marked for retry parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/52cac3f5-dbc4-475d-b6b6-bae04ea6ee76_characterizeImage_855911_31.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/52cac3f5-dbc4-475d-b6b6-bae04ea6ee76_characterizeImage_855911_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash 3214a44fb0bc6e3a79673fbcc3396199 parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec22fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 106 try 1 launched on executor multi with executor id 137 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/52cac3f5-dbc4-475d-b6b6-bae04ea6ee76_characterizeImage_855911_31.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/52cac3f5-dbc4-475d-b6b6-bae04ea6ee76_characterizeImage_855911_31.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 20, active_tasks = 77 parsl.dataflow.strategy DEBUG: Executor multi has 77 active tasks, 20/0 running/pending blocks, and 21 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22667511342224 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 132 try 0 failed parsl.dataflow.dflow INFO: Task 132 marked for retry parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855821/4c2d509c-480b-4b38-8207-7abd607c2208_characterizeImage_855821_2.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855821/4c2d509c-480b-4b38-8207-7abd607c2208_characterizeImage_855821_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 132 has memoization hash b2f76cb283beaf8b0d7e287f6c52b1f4 parsl.dataflow.memoization INFO: Task 132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec74680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 132 try 1 launched on executor multi with executor id 138 parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855821/4c2d509c-480b-4b38-8207-7abd607c2208_characterizeImage_855821_2.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855821/4c2d509c-480b-4b38-8207-7abd607c2208_characterizeImage_855821_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 108 try 0 failed parsl.dataflow.dflow INFO: Task 108 marked for retry parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/ab699881-3e42-4cd7-aaea-c25fb4618495_characterizeImage_855911_17.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/ab699881-3e42-4cd7-aaea-c25fb4618495_characterizeImage_855911_17.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 108 has memoization hash 9d8f97b8e1cde209a2efeffc8ae6480a parsl.dataflow.memoization INFO: Task 108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec22de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 108 try 1 launched on executor multi with executor id 139 parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/ab699881-3e42-4cd7-aaea-c25fb4618495_characterizeImage_855911_17.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/ab699881-3e42-4cd7-aaea-c25fb4618495_characterizeImage_855911_17.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 104 try 0 failed parsl.dataflow.dflow INFO: Task 104 marked for retry parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/98e11768-2250-42f8-90b4-f807d804f98c_characterizeImage_855910_25.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/98e11768-2250-42f8-90b4-f807d804f98c_characterizeImage_855910_25.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 104 has memoization hash 04c191a1fe343b963abedb882721781f parsl.dataflow.memoization INFO: Task 104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec22c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 104 try 1 launched on executor multi with executor id 140 parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/98e11768-2250-42f8-90b4-f807d804f98c_characterizeImage_855910_25.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/98e11768-2250-42f8-90b4-f807d804f98c_characterizeImage_855910_25.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 120 try 0 failed parsl.dataflow.dflow INFO: Task 120 marked for retry parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/557817d1-e506-4563-a201-83fb89cacc81_characterizeImage_855910_20.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/557817d1-e506-4563-a201-83fb89cacc81_characterizeImage_855910_20.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 120 has memoization hash 7e22f0114c0ed233b4b693665ac93125 parsl.dataflow.memoization INFO: Task 120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec23b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 120 try 1 launched on executor multi with executor id 141 parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/557817d1-e506-4563-a201-83fb89cacc81_characterizeImage_855910_20.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/557817d1-e506-4563-a201-83fb89cacc81_characterizeImage_855910_20.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 114 try 0 failed parsl.dataflow.dflow INFO: Task 114 marked for retry parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/5ac1ce51-aedf-4f82-9db8-8ec41a8abf75_characterizeImage_855911_4.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/5ac1ce51-aedf-4f82-9db8-8ec41a8abf75_characterizeImage_855911_4.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 114 has memoization hash c27671771e741ba6ba212498da98a89a parsl.dataflow.memoization INFO: Task 114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec23560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 114 try 1 launched on executor multi with executor id 142 parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/5ac1ce51-aedf-4f82-9db8-8ec41a8abf75_characterizeImage_855911_4.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/5ac1ce51-aedf-4f82-9db8-8ec41a8abf75_characterizeImage_855911_4.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 124 try 0 failed parsl.dataflow.dflow INFO: Task 124 marked for retry parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/76337c37-3ebd-49be-b1db-586b608487b7_characterizeImage_855911_20.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/76337c37-3ebd-49be-b1db-586b608487b7_characterizeImage_855911_20.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 124 has memoization hash 783324ae5e67dbaeabe63595c1dd6d5e parsl.dataflow.memoization INFO: Task 124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec74040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 124 try 1 launched on executor multi with executor id 143 parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/76337c37-3ebd-49be-b1db-586b608487b7_characterizeImage_855911_20.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/76337c37-3ebd-49be-b1db-586b608487b7_characterizeImage_855911_20.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 116 try 0 failed parsl.dataflow.dflow INFO: Task 116 marked for retry parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/ba6bd293-3d10-4464-ba8d-d203f2b41a51_characterizeImage_855911_46.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/ba6bd293-3d10-4464-ba8d-d203f2b41a51_characterizeImage_855911_46.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 116 has memoization hash b8841e91316276e5dd6891d38acb698c parsl.dataflow.memoization INFO: Task 116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec23920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 116 try 1 launched on executor multi with executor id 144 parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/ba6bd293-3d10-4464-ba8d-d203f2b41a51_characterizeImage_855911_46.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/ba6bd293-3d10-4464-ba8d-d203f2b41a51_characterizeImage_855911_46.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 118 try 0 failed parsl.dataflow.dflow INFO: Task 118 marked for retry parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/3f32effb-4971-4cd6-a6fb-07c39cefa4e0_characterizeImage_855910_33.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/3f32effb-4971-4cd6-a6fb-07c39cefa4e0_characterizeImage_855910_33.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash 6d57f783396e1d260a5a5ee9cd4bdb2c parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec23740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 1 launched on executor multi with executor id 145 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/3f32effb-4971-4cd6-a6fb-07c39cefa4e0_characterizeImage_855910_33.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/3f32effb-4971-4cd6-a6fb-07c39cefa4e0_characterizeImage_855910_33.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 20, active_tasks = 77 parsl.dataflow.strategy DEBUG: Executor multi has 77 active tasks, 20/0 running/pending blocks, and 21 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22667511342224 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 126 try 0 failed parsl.dataflow.dflow INFO: Task 126 marked for retry parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/75228b8e-af24-4ef7-a2ab-6ad77397b1f6_characterizeImage_855910_39.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/75228b8e-af24-4ef7-a2ab-6ad77397b1f6_characterizeImage_855910_39.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 126 has memoization hash 6b781062ea895e02e9694ec3fd22d6dd parsl.dataflow.memoization INFO: Task 126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec742c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 126 try 1 launched on executor multi with executor id 146 parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/75228b8e-af24-4ef7-a2ab-6ad77397b1f6_characterizeImage_855910_39.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/75228b8e-af24-4ef7-a2ab-6ad77397b1f6_characterizeImage_855910_39.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 130 try 0 failed parsl.dataflow.dflow INFO: Task 130 marked for retry parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/530393c1-16cf-4683-9451-8f61161fb9ab_characterizeImage_855911_33.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/530393c1-16cf-4683-9451-8f61161fb9ab_characterizeImage_855911_33.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 130 has memoization hash 4f1947c73e07e6a03abda598178896eb parsl.dataflow.memoization INFO: Task 130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec744a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 130 try 1 launched on executor multi with executor id 147 parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/530393c1-16cf-4683-9451-8f61161fb9ab_characterizeImage_855911_33.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/530393c1-16cf-4683-9451-8f61161fb9ab_characterizeImage_855911_33.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 122 try 0 failed parsl.dataflow.dflow INFO: Task 122 marked for retry parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/b24378e6-0128-4e39-8954-9690a15457bd_characterizeImage_855911_47.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/b24378e6-0128-4e39-8954-9690a15457bd_characterizeImage_855911_47.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 122 has memoization hash f08c8251214a6eb92bf951472fb4501e parsl.dataflow.memoization INFO: Task 122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec23ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 122 try 1 launched on executor multi with executor id 148 parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/b24378e6-0128-4e39-8954-9690a15457bd_characterizeImage_855911_47.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/b24378e6-0128-4e39-8954-9690a15457bd_characterizeImage_855911_47.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 78 try 0 failed parsl.dataflow.dflow INFO: Task 78 marked for retry parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/1d3d4c4e-0c48-4edb-8599-e62a2b871238_characterizeImage_855910_61.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/1d3d4c4e-0c48-4edb-8599-e62a2b871238_characterizeImage_855910_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 78 has memoization hash 20dbd507b90cddb48c5bec60f75cd595 parsl.dataflow.memoization INFO: Task 78 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec211c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 78 try 1 launched on executor multi with executor id 149 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/1d3d4c4e-0c48-4edb-8599-e62a2b871238_characterizeImage_855910_61.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/1d3d4c4e-0c48-4edb-8599-e62a2b871238_characterizeImage_855910_61.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 128 try 0 failed parsl.dataflow.dflow INFO: Task 128 marked for retry parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/b9507bfd-4907-420b-89e8-49e3d290e347_characterizeImage_855910_13.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/b9507bfd-4907-420b-89e8-49e3d290e347_characterizeImage_855910_13.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 128 has memoization hash 547525b8c5361f46105c6721959871a8 parsl.dataflow.memoization INFO: Task 128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec740e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 128 try 1 launched on executor multi with executor id 150 parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/b9507bfd-4907-420b-89e8-49e3d290e347_characterizeImage_855910_13.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/b9507bfd-4907-420b-89e8-49e3d290e347_characterizeImage_855910_13.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 138 try 0 failed parsl.dataflow.dflow INFO: Task 138 marked for retry parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/7ed6d274-7055-4b37-9f85-83443fae8a1f_characterizeImage_855911_8.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/7ed6d274-7055-4b37-9f85-83443fae8a1f_characterizeImage_855911_8.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 138 has memoization hash 8c00d28bc925c30b332208303fa03956 parsl.dataflow.memoization INFO: Task 138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec74a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 138 try 1 launched on executor multi with executor id 151 parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/7ed6d274-7055-4b37-9f85-83443fae8a1f_characterizeImage_855911_8.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/7ed6d274-7055-4b37-9f85-83443fae8a1f_characterizeImage_855911_8.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 136 try 0 failed parsl.dataflow.dflow INFO: Task 136 marked for retry parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/0c0696ad-bcb9-4ee0-bd31-858ea5955eee_characterizeImage_855911_19.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/0c0696ad-bcb9-4ee0-bd31-858ea5955eee_characterizeImage_855911_19.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 136 has memoization hash 85c07d4797533be7aacba981040dfd08 parsl.dataflow.memoization INFO: Task 136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec74c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 136 try 1 launched on executor multi with executor id 152 parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/0c0696ad-bcb9-4ee0-bd31-858ea5955eee_characterizeImage_855911_19.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/0c0696ad-bcb9-4ee0-bd31-858ea5955eee_characterizeImage_855911_19.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 134 try 0 failed parsl.dataflow.dflow INFO: Task 134 marked for retry parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/1ac43a9a-71de-4903-bf82-a267193afbc2_characterizeImage_855911_53.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/1ac43a9a-71de-4903-bf82-a267193afbc2_characterizeImage_855911_53.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 134 has memoization hash f4b087fca69cbd10e0ea212117e8ab18 parsl.dataflow.memoization INFO: Task 134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec74860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 134 try 1 launched on executor multi with executor id 153 parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/1ac43a9a-71de-4903-bf82-a267193afbc2_characterizeImage_855911_53.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/1ac43a9a-71de-4903-bf82-a267193afbc2_characterizeImage_855911_53.stderr 2024-03-13 01:50:46 proc_lsst.multi:146 [INFO] found job 23564 in provider local proc_lsst.multi INFO: found job 23564 in provider local 2024-03-13 01:50:47 proc_lsst.multi:146 [INFO] found job 17006978 in provider astro proc_lsst.multi INFO: found job 17006978 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:50:47 proc_lsst.multi:146 [INFO] found job 17006979 in provider astro proc_lsst.multi INFO: found job 17006979 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:50:48 proc_lsst.multi:146 [INFO] found job 17006980 in provider astro proc_lsst.multi INFO: found job 17006980 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:50:48 proc_lsst.multi:146 [INFO] found job 17006981 in provider astro proc_lsst.multi INFO: found job 17006981 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 1 try 1 failed parsl.dataflow.dflow ERROR: Task 1 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 945eb2730401d3236a5cb33f0270457a with result from task 1 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/846708ef-297c-4241-80c4-81276b622400_calibrate_855911_9.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/846708ef-297c-4241-80c4-81276b622400_calibrate_855911_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:50:49 proc_lsst.multi:146 [INFO] found job 17006982 in provider astro proc_lsst.multi INFO: found job 17006982 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 3 try 1 failed parsl.dataflow.dflow ERROR: Task 3 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 8060fa9cd95588ec1f90daf399058a28 with result from task 3 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/fc6e7088-7176-4422-96f7-861be6218102_calibrate_855911_14.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/fc6e7088-7176-4422-96f7-861be6218102_calibrate_855911_14.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 16 try 1 failed parsl.dataflow.dflow ERROR: Task 16 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry f919b0be28762e72b432c44f5167d861 with result from task 16 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 17 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 17 try 0 failed parsl.dataflow.dflow INFO: Task 17 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/a831b403-d444-4c5b-a208-4be861f36723_calibrate_855910_16.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/a831b403-d444-4c5b-a208-4be861f36723_calibrate_855910_16.stderr parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/9390e644-3687-46b6-aa37-eb7c80d4de29_characterizeImage_855910_16.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/9390e644-3687-46b6-aa37-eb7c80d4de29_characterizeImage_855910_16.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 112 try 0 failed parsl.dataflow.dflow INFO: Task 112 marked for retry parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/98714589-92e8-4855-973f-8251c4b5f8c9_characterizeImage_855911_61.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/98714589-92e8-4855-973f-8251c4b5f8c9_characterizeImage_855911_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 112 has memoization hash e2666f3d8b9748f98ae027abbca45960 parsl.dataflow.memoization INFO: Task 112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x149daec23380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 112 try 1 launched on executor multi with executor id 154 parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/98714589-92e8-4855-973f-8251c4b5f8c9_characterizeImage_855911_61.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/98714589-92e8-4855-973f-8251c4b5f8c9_characterizeImage_855911_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:50:49 proc_lsst.multi:146 [INFO] found job 17006983 in provider astro proc_lsst.multi INFO: found job 17006983 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:50:49 proc_lsst.multi:146 [INFO] found job 17006984 in provider astro proc_lsst.multi INFO: found job 17006984 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:50:50 proc_lsst.multi:146 [INFO] found job 17006985 in provider astro proc_lsst.multi INFO: found job 17006985 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:50:50 proc_lsst.multi:146 [INFO] found job 17006986 in provider astro proc_lsst.multi INFO: found job 17006986 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:50:51 proc_lsst.multi:146 [INFO] found job 17006987 in provider astro proc_lsst.multi INFO: found job 17006987 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:50:51 proc_lsst.multi:146 [INFO] found job 17006988 in provider astro proc_lsst.multi INFO: found job 17006988 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 2 try 1 failed parsl.dataflow.dflow ERROR: Task 2 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry f4bf0c3c4eedcb763ff4ab113b3e09fe with result from task 2 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/1ba9f290-426e-40fd-b776-5140f06d5856_calibrate_855911_32.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/1ba9f290-426e-40fd-b776-5140f06d5856_calibrate_855911_32.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 24 try 1 failed parsl.dataflow.dflow ERROR: Task 24 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 843374210022ec35e6ac2113ed5276ce with result from task 24 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 25 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 25 try 0 failed parsl.dataflow.dflow INFO: Task 25 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/aca89bf7-8aaa-4518-b63f-8e885701136b_calibrate_855911_58.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/aca89bf7-8aaa-4518-b63f-8e885701136b_calibrate_855911_58.stderr parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/6175499a-5177-4137-b42d-38fc1c31a263_characterizeImage_855911_58.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/6175499a-5177-4137-b42d-38fc1c31a263_characterizeImage_855911_58.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 14 try 1 failed parsl.dataflow.dflow ERROR: Task 14 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 5d5e14c03ea0cf657543b8fa137794fd with result from task 14 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 15 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 15 try 0 failed parsl.dataflow.dflow INFO: Task 15 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/8e6e6626-6b5d-4941-ba4b-87afac47e63e_calibrate_855911_38.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/8e6e6626-6b5d-4941-ba4b-87afac47e63e_calibrate_855911_38.stderr parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/08082892-9973-49b4-81e5-e3b332deac5b_characterizeImage_855911_38.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/08082892-9973-49b4-81e5-e3b332deac5b_characterizeImage_855911_38.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 26 try 1 failed parsl.dataflow.dflow ERROR: Task 26 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 1470caf28784b09d64c5bf305806b824 with result from task 26 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 27 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 27 try 0 failed parsl.dataflow.dflow INFO: Task 27 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/93d3f944-48cb-4aeb-ab30-d372efb0121c_calibrate_855911_5.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/93d3f944-48cb-4aeb-ab30-d372efb0121c_calibrate_855911_5.stderr parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/1a97605c-6432-4995-b290-68cb12b763a2_characterizeImage_855911_5.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/1a97605c-6432-4995-b290-68cb12b763a2_characterizeImage_855911_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:50:52 proc_lsst.multi:146 [INFO] found job 17006989 in provider astro proc_lsst.multi INFO: found job 17006989 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 22 try 1 failed parsl.dataflow.dflow ERROR: Task 22 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 9f73ec5b9a7aef9a54c21b04299c3355 with result from task 22 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 23 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 23 try 0 failed parsl.dataflow.dflow INFO: Task 23 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/e5011698-c656-406a-98a9-5bb1b6fd028c_calibrate_855910_46.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/e5011698-c656-406a-98a9-5bb1b6fd028c_calibrate_855910_46.stderr parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/800a2922-f721-4d25-822c-a7094ee8caed_characterizeImage_855910_46.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/800a2922-f721-4d25-822c-a7094ee8caed_characterizeImage_855910_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:50:52 proc_lsst.multi:146 [INFO] found job 17006990 in provider astro proc_lsst.multi INFO: found job 17006990 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:50:53 proc_lsst.multi:146 [INFO] found job 17006991 in provider astro proc_lsst.multi INFO: found job 17006991 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 20 try 1 failed parsl.dataflow.dflow ERROR: Task 20 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 7cdaea0c83a308f4fe22ba2e71c4ee16 with result from task 20 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 21 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 21 try 0 failed parsl.dataflow.dflow INFO: Task 21 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/70a6a4e3-b03e-4e7c-b26c-911348278962_calibrate_855911_15.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/70a6a4e3-b03e-4e7c-b26c-911348278962_calibrate_855911_15.stderr parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/ca786ff8-5ae7-4c4c-845b-c815ba80c3a2_characterizeImage_855911_15.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/ca786ff8-5ae7-4c4c-845b-c815ba80c3a2_characterizeImage_855911_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:50:53 proc_lsst.multi:146 [INFO] found job 17006992 in provider astro proc_lsst.multi INFO: found job 17006992 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 48 try 1 failed parsl.dataflow.dflow ERROR: Task 48 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 7ee445c26b0e81f76caf54fd1e2daa1c with result from task 48 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 49 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 49 try 0 failed parsl.dataflow.dflow INFO: Task 49 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855909/bee013aa-2986-403d-b6d5-227b402f581e_calibrate_855909_2.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855909/bee013aa-2986-403d-b6d5-227b402f581e_calibrate_855909_2.stderr parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855909/d4097b91-b153-43b0-866d-30e0c77604e5_characterizeImage_855909_2.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855909/d4097b91-b153-43b0-866d-30e0c77604e5_characterizeImage_855909_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:50:54 proc_lsst.multi:146 [INFO] found job 17006993 in provider astro proc_lsst.multi INFO: found job 17006993 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:50:54 proc_lsst.multi:146 [INFO] found job 17006994 in provider astro proc_lsst.multi INFO: found job 17006994 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:50:55 proc_lsst.multi:146 [INFO] found job 17006995 in provider astro proc_lsst.multi INFO: found job 17006995 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 52 try 1 failed parsl.dataflow.dflow ERROR: Task 52 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 12054b167c7a649a7aaacf6e663f5fac with result from task 52 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 53 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 53 try 0 failed parsl.dataflow.dflow INFO: Task 53 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855820/76347c0e-2f50-4e9f-83da-da3ca61ef1bc_calibrate_855820_2.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855820/76347c0e-2f50-4e9f-83da-da3ca61ef1bc_calibrate_855820_2.stderr parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855820/e4f80ceb-3483-453a-aac6-2222e463e2a6_characterizeImage_855820_2.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855820/e4f80ceb-3483-453a-aac6-2222e463e2a6_characterizeImage_855820_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:50:55 proc_lsst.multi:146 [INFO] found job 17006996 in provider astro proc_lsst.multi INFO: found job 17006996 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 20, active_tasks = 66 parsl.dataflow.strategy DEBUG: Executor multi has 66 active tasks, 20/0 running/pending blocks, and 21 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22667511342224 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 20, active_tasks = 66 parsl.dataflow.strategy DEBUG: Executor multi has 66 active tasks, 20/0 running/pending blocks, and 21 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22667511342224 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 28 try 1 failed parsl.dataflow.dflow ERROR: Task 28 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 8aafad68ee30a513d7ff0fc7ce1f5c26 with result from task 28 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 29 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 29 try 0 failed parsl.dataflow.dflow INFO: Task 29 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/b57923bc-2ec4-4aa6-802a-b1b64c2ac3d3_calibrate_855911_39.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/b57923bc-2ec4-4aa6-802a-b1b64c2ac3d3_calibrate_855911_39.stderr parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/983a01f8-c7bf-4fef-84c8-a8caf58c74b6_characterizeImage_855911_39.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/983a01f8-c7bf-4fef-84c8-a8caf58c74b6_characterizeImage_855911_39.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 18 try 1 failed parsl.dataflow.dflow ERROR: Task 18 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry bb34b92b8d6ed10882635aeefc65760a with result from task 18 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 19 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 19 try 0 failed parsl.dataflow.dflow INFO: Task 19 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/1ef4843d-2ec4-412d-80d5-0ce05845241f_calibrate_855911_13.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/1ef4843d-2ec4-412d-80d5-0ce05845241f_calibrate_855911_13.stderr parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/a090aba4-4a38-4573-a572-f5cabab5b056_characterizeImage_855911_13.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/a090aba4-4a38-4573-a572-f5cabab5b056_characterizeImage_855911_13.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 6 try 1 failed parsl.dataflow.dflow ERROR: Task 6 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry d6db8b96e78af2a4f34530c3c84023ad with result from task 6 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/634fa457-accc-4576-90ce-3d577bc12af2_calibrate_855911_25.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/634fa457-accc-4576-90ce-3d577bc12af2_calibrate_855911_25.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 0 try 1 failed parsl.dataflow.dflow ERROR: Task 0 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry c877fe17b3afe9eaa5084e2b9088ce24 with result from task 0 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/19a20ea9-58db-4c2b-9e4e-82d1cf7cb8c9_calibrate_855911_35.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/19a20ea9-58db-4c2b-9e4e-82d1cf7cb8c9_calibrate_855911_35.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 11 try 1 failed parsl.dataflow.dflow ERROR: Task 11 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 1f0a16be7fea9254b03f305974f9a281 with result from task 11 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/13eadaed-b6ed-46a9-a1fe-1e9790987a07_calibrate_855910_21.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/13eadaed-b6ed-46a9-a1fe-1e9790987a07_calibrate_855910_21.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 8 try 1 failed parsl.dataflow.dflow ERROR: Task 8 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 5be10890c0174fc8f64aad9dfabce424 with result from task 8 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/63247ced-0399-4767-bf05-e6d64abbfb8d_calibrate_855910_9.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/63247ced-0399-4767-bf05-e6d64abbfb8d_calibrate_855910_9.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 13 try 1 failed parsl.dataflow.dflow ERROR: Task 13 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry e4cdb6332ecaf44719ea625b51e0c07e with result from task 13 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/8784752d-bf08-47df-b8a0-c97ff7f9b4a0_calibrate_855910_4.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/8784752d-bf08-47df-b8a0-c97ff7f9b4a0_calibrate_855910_4.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 20, active_tasks = 59 parsl.dataflow.strategy DEBUG: Executor multi has 59 active tasks, 20/0 running/pending blocks, and 21 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22667511342224 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 30 try 1 failed parsl.dataflow.dflow ERROR: Task 30 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry c8f9f8853974013de48d830f921f325c with result from task 30 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 31 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 31 try 0 failed parsl.dataflow.dflow INFO: Task 31 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/292f5b55-fee2-4bad-af5a-ff6f93769949_calibrate_855911_11.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/292f5b55-fee2-4bad-af5a-ff6f93769949_calibrate_855911_11.stderr parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/5c0d4d51-2440-430b-9231-d536d662efe7_characterizeImage_855911_11.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/5c0d4d51-2440-430b-9231-d536d662efe7_characterizeImage_855911_11.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 5 try 1 failed parsl.dataflow.dflow ERROR: Task 5 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 7b9f97e5556c134f7b40ed88f2184075 with result from task 5 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/db2895de-bf84-4a3e-8579-ea13eb770871_calibrate_855911_34.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/db2895de-bf84-4a3e-8579-ea13eb770871_calibrate_855911_34.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 9 try 1 failed parsl.dataflow.dflow ERROR: Task 9 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 62215dcfe5e4e0955f0641f249b0e10d with result from task 9 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/8b2fa882-73b1-4798-923c-386c396bac28_calibrate_855911_27.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/8b2fa882-73b1-4798-923c-386c396bac28_calibrate_855911_27.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 42 try 1 failed parsl.dataflow.dflow ERROR: Task 42 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry dd11e2e4560735a94dba9365e9b48f75 with result from task 42 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 43 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 43 try 0 failed parsl.dataflow.dflow INFO: Task 43 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/17fe73b1-3609-4d1c-9f52-f77f49196b11_calibrate_855911_3.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/17fe73b1-3609-4d1c-9f52-f77f49196b11_calibrate_855911_3.stderr parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/8774d934-8e13-4ee7-9af4-b8633afb8117_characterizeImage_855911_3.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/8774d934-8e13-4ee7-9af4-b8633afb8117_characterizeImage_855911_3.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 10 try 1 failed parsl.dataflow.dflow ERROR: Task 10 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 569c5fab366f4d0989df9b03d78cace5 with result from task 10 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/89d5f5c0-6cde-42f9-9981-aa0a6946ff66_calibrate_855910_34.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/89d5f5c0-6cde-42f9-9981-aa0a6946ff66_calibrate_855910_34.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 38 try 1 failed parsl.dataflow.dflow ERROR: Task 38 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 279fc759a984b5b6d09ebe68191901a9 with result from task 38 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 39 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 39 try 0 failed parsl.dataflow.dflow INFO: Task 39 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/8a7f14cc-ca07-4ee3-99f3-1d0a00998ca7_calibrate_855911_1.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/8a7f14cc-ca07-4ee3-99f3-1d0a00998ca7_calibrate_855911_1.stderr parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/182ff692-3216-4bc9-a828-1736d1daf016_characterizeImage_855911_1.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/182ff692-3216-4bc9-a828-1736d1daf016_characterizeImage_855911_1.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 40 try 1 failed parsl.dataflow.dflow ERROR: Task 40 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 695315fd2af3281397c8bf1391395ac0 with result from task 40 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 41 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 41 try 0 failed parsl.dataflow.dflow INFO: Task 41 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/4acfb2b2-cdb5-4975-89d8-81bf90bb7c62_calibrate_855910_47.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/4acfb2b2-cdb5-4975-89d8-81bf90bb7c62_calibrate_855910_47.stderr parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/c82fb047-b7dc-49ad-845d-7d9c7cc1ecd1_characterizeImage_855910_47.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/c82fb047-b7dc-49ad-845d-7d9c7cc1ecd1_characterizeImage_855910_47.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 36 try 1 failed parsl.dataflow.dflow ERROR: Task 36 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 3e33fcbdba0efe15fcc1181bee9baaaf with result from task 36 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 37 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 37 try 0 failed parsl.dataflow.dflow INFO: Task 37 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/eceba4c3-53d6-441d-bab4-4eb9bd4924c4_calibrate_855911_21.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/eceba4c3-53d6-441d-bab4-4eb9bd4924c4_calibrate_855911_21.stderr parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/256f3f22-a5e3-4dc6-bfbf-67a9cd648029_characterizeImage_855911_21.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/256f3f22-a5e3-4dc6-bfbf-67a9cd648029_characterizeImage_855911_21.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 34 try 1 failed parsl.dataflow.dflow ERROR: Task 34 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 075cfe43ceab10b0e0991f5e1ce3038b with result from task 34 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 35 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 35 try 0 failed parsl.dataflow.dflow INFO: Task 35 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/aa08cce6-4c1a-4c21-ae8d-89e84aa88233_calibrate_855911_16.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/aa08cce6-4c1a-4c21-ae8d-89e84aa88233_calibrate_855911_16.stderr parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/4f91a3bb-dde6-45ad-ae80-2cec03be0edf_characterizeImage_855911_16.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/4f91a3bb-dde6-45ad-ae80-2cec03be0edf_characterizeImage_855911_16.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 66 try 1 failed parsl.dataflow.dflow ERROR: Task 66 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 6b1738bbc37c2be15f7810fb8d8af598 with result from task 66 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 67 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 67 try 0 failed parsl.dataflow.dflow INFO: Task 67 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855822/eb5a012e-803e-4612-8981-d26da30c77f6_calibrate_855822_2.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855822/eb5a012e-803e-4612-8981-d26da30c77f6_calibrate_855822_2.stderr parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855822/ea0e240a-012a-4c18-88ba-5ff6db71a5ef_characterizeImage_855822_2.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855822/ea0e240a-012a-4c18-88ba-5ff6db71a5ef_characterizeImage_855822_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 32 try 1 failed parsl.dataflow.dflow ERROR: Task 32 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry a38cb225b314a6deb4f26a38c497b849 with result from task 32 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 33 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 33 try 0 failed parsl.dataflow.dflow INFO: Task 33 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/fc7f5758-1da1-42b4-b0a5-3ad3030c8ce3_calibrate_855910_27.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/fc7f5758-1da1-42b4-b0a5-3ad3030c8ce3_calibrate_855910_27.stderr parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/8d6c5ccf-e869-40b5-8751-c8d8d42c408d_characterizeImage_855910_27.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/8d6c5ccf-e869-40b5-8751-c8d8d42c408d_characterizeImage_855910_27.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 20, active_tasks = 48 parsl.dataflow.strategy DEBUG: Executor multi has 48 active tasks, 20/0 running/pending blocks, and 21 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22667511342224 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 56 try 1 failed parsl.dataflow.dflow ERROR: Task 56 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 2d4744912aceb1810e9243c1b43cc8d4 with result from task 56 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 57 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 57 try 0 failed parsl.dataflow.dflow INFO: Task 57 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/8b65e254-8651-4958-8580-8863d2865d0b_calibrate_855911_28.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/8b65e254-8651-4958-8580-8863d2865d0b_calibrate_855911_28.stderr parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/66bb60c0-5531-4492-b309-e5bd6388fe1d_characterizeImage_855911_28.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/66bb60c0-5531-4492-b309-e5bd6388fe1d_characterizeImage_855911_28.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 50 try 1 failed parsl.dataflow.dflow ERROR: Task 50 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry c81f729b57ed59987b5c33c48e4370ad with result from task 50 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 51 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 51 try 0 failed parsl.dataflow.dflow INFO: Task 51 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/3d8fbd9d-2437-4563-b087-471fdead2e44_calibrate_855911_48.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/3d8fbd9d-2437-4563-b087-471fdead2e44_calibrate_855911_48.stderr parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/0a7de8f6-9d73-4bcc-9593-33726fcaba42_characterizeImage_855911_48.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/0a7de8f6-9d73-4bcc-9593-33726fcaba42_characterizeImage_855911_48.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 44 try 1 failed parsl.dataflow.dflow ERROR: Task 44 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 3bff9b078737f4a7145ba0c4f60a220c with result from task 44 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 45 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 45 try 0 failed parsl.dataflow.dflow INFO: Task 45 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/6630661e-76c6-419a-b14a-65320d334ec8_calibrate_855910_51.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/6630661e-76c6-419a-b14a-65320d334ec8_calibrate_855910_51.stderr parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/22235225-dfff-445b-b762-eec54521fba4_characterizeImage_855910_51.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/22235225-dfff-445b-b762-eec54521fba4_characterizeImage_855910_51.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 58 try 1 failed parsl.dataflow.dflow ERROR: Task 58 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 4ca04ffbdd3ef0e35306378e0ba39cfb with result from task 58 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 59 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 59 try 0 failed parsl.dataflow.dflow INFO: Task 59 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/4dac306a-d17e-4f60-adec-4ad7371c03e1_calibrate_855911_18.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/4dac306a-d17e-4f60-adec-4ad7371c03e1_calibrate_855911_18.stderr parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/5cabd763-dbeb-4e4d-a653-456299812857_characterizeImage_855911_18.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/5cabd763-dbeb-4e4d-a653-456299812857_characterizeImage_855911_18.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 96 try 1 failed parsl.dataflow.dflow ERROR: Task 96 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 31effd051c9b026c54abed0e44d9d24f with result from task 96 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 97 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 97 try 0 failed parsl.dataflow.dflow INFO: Task 97 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/4613434f-e188-467d-a9a2-f666df2cc983_calibrate_855911_2.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/4613434f-e188-467d-a9a2-f666df2cc983_calibrate_855911_2.stderr parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/880f73cb-49c4-4049-b1b9-ca8ab2091500_characterizeImage_855911_2.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/880f73cb-49c4-4049-b1b9-ca8ab2091500_characterizeImage_855911_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 46 try 1 failed parsl.dataflow.dflow ERROR: Task 46 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry ef41369a22e73637504962f6c442c39b with result from task 46 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 47 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 47 try 0 failed parsl.dataflow.dflow INFO: Task 47 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/7db034de-b54f-4e38-b72e-847fd053044c_calibrate_855911_22.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/7db034de-b54f-4e38-b72e-847fd053044c_calibrate_855911_22.stderr parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/640eda07-0934-42c3-a791-049094bd785a_characterizeImage_855911_22.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/640eda07-0934-42c3-a791-049094bd785a_characterizeImage_855911_22.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 4 try 1 failed parsl.dataflow.dflow ERROR: Task 4 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry f5dc0abcdf5fdb8a5a6b8ffcfc7941a0 with result from task 4 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/64c54979-dfdb-429a-897d-cbd231b11b96_calibrate_855911_59.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/64c54979-dfdb-429a-897d-cbd231b11b96_calibrate_855911_59.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 62 try 1 failed parsl.dataflow.dflow ERROR: Task 62 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 5df1050a7ee8ebc372b04620571f0fc6 with result from task 62 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 63 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 63 try 0 failed parsl.dataflow.dflow INFO: Task 63 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/036e8f06-a41f-4e6e-bc30-2172781e55f7_calibrate_855910_26.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/036e8f06-a41f-4e6e-bc30-2172781e55f7_calibrate_855910_26.stderr parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/529cfa1c-af53-4ac8-99ff-24ac46e84a8c_characterizeImage_855910_26.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/529cfa1c-af53-4ac8-99ff-24ac46e84a8c_characterizeImage_855910_26.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 20, active_tasks = 40 parsl.dataflow.strategy DEBUG: Executor multi has 40 active tasks, 20/0 running/pending blocks, and 21 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22667511342224 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 54 try 1 failed parsl.dataflow.dflow ERROR: Task 54 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 194c1e7d0b4f56cf4fa0a12795269d4e with result from task 54 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 55 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 55 try 0 failed parsl.dataflow.dflow INFO: Task 55 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/bd2bb832-b795-479c-9a2e-25afeed843db_calibrate_855910_1.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/bd2bb832-b795-479c-9a2e-25afeed843db_calibrate_855910_1.stderr parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/c73aa34e-5b41-499a-b16f-657e63b319c2_characterizeImage_855910_1.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/c73aa34e-5b41-499a-b16f-657e63b319c2_characterizeImage_855910_1.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 64 try 1 failed parsl.dataflow.dflow ERROR: Task 64 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry a3b8a455c12f326ea503d6fdfc377777 with result from task 64 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 65 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 65 try 0 failed parsl.dataflow.dflow INFO: Task 65 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/09b3f152-001b-4c1f-8d05-81ae43944dbd_calibrate_855911_12.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/09b3f152-001b-4c1f-8d05-81ae43944dbd_calibrate_855911_12.stderr parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/5cab9c4c-303d-4525-8e58-ae2196b669a9_characterizeImage_855911_12.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/5cab9c4c-303d-4525-8e58-ae2196b669a9_characterizeImage_855911_12.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 60 try 1 failed parsl.dataflow.dflow ERROR: Task 60 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 841e65085f790d02600b13fc682951a5 with result from task 60 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 61 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 61 try 0 failed parsl.dataflow.dflow INFO: Task 61 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/28b44b08-1116-4b40-aeec-d132725c87a8_calibrate_855910_14.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/28b44b08-1116-4b40-aeec-d132725c87a8_calibrate_855910_14.stderr parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/f090aa20-a60f-4eb2-a122-cac87a55d70b_characterizeImage_855910_14.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/f090aa20-a60f-4eb2-a122-cac87a55d70b_characterizeImage_855910_14.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 74 try 1 failed parsl.dataflow.dflow ERROR: Task 74 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry b55d017ba070d9dfacdfbfe16b04c232 with result from task 74 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 75 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 75 try 0 failed parsl.dataflow.dflow INFO: Task 75 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/e85548c1-bfb4-4f95-8039-fbfb1feca6a8_calibrate_855911_45.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/e85548c1-bfb4-4f95-8039-fbfb1feca6a8_calibrate_855911_45.stderr parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/1ec0a70e-4d13-451d-93b7-bb739eb96816_characterizeImage_855911_45.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/1ec0a70e-4d13-451d-93b7-bb739eb96816_characterizeImage_855911_45.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 72 try 1 failed parsl.dataflow.dflow ERROR: Task 72 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 7aae0460c0c17a98effb8d617b8de6f8 with result from task 72 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 73 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 73 try 0 failed parsl.dataflow.dflow INFO: Task 73 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/f1e05433-927b-4b14-a780-70f1f5bedf4f_calibrate_855911_6.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/f1e05433-927b-4b14-a780-70f1f5bedf4f_calibrate_855911_6.stderr parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/604d91d5-c52a-4c29-a960-33f2108dbba4_characterizeImage_855911_6.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/604d91d5-c52a-4c29-a960-33f2108dbba4_characterizeImage_855911_6.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 70 try 1 failed parsl.dataflow.dflow ERROR: Task 70 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 33e26fbdc1fbf2088ecc6cf991dd7652 with result from task 70 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 71 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 71 try 0 failed parsl.dataflow.dflow INFO: Task 71 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/b65146fa-f545-4dc3-acda-ef9353e8b57d_calibrate_855910_58.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/b65146fa-f545-4dc3-acda-ef9353e8b57d_calibrate_855910_58.stderr parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/1c3713cf-9f09-41c4-a287-2ef9026262da_characterizeImage_855910_58.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/1c3713cf-9f09-41c4-a287-2ef9026262da_characterizeImage_855910_58.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 80 try 1 failed parsl.dataflow.dflow ERROR: Task 80 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry cec517821c7f0321270b94ac5c05e9ff with result from task 80 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 81 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 81 try 0 failed parsl.dataflow.dflow INFO: Task 81 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/9cf622a9-01dc-44b1-a18b-d67e28089b5f_calibrate_855910_32.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/9cf622a9-01dc-44b1-a18b-d67e28089b5f_calibrate_855910_32.stderr parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/8e9e637d-c542-4349-9402-9cb673ae28b0_characterizeImage_855910_32.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/8e9e637d-c542-4349-9402-9cb673ae28b0_characterizeImage_855910_32.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 20, active_tasks = 33 parsl.dataflow.strategy DEBUG: Executor multi has 33 active tasks, 20/0 running/pending blocks, and 21 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22667511342224 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 68 try 1 failed parsl.dataflow.dflow ERROR: Task 68 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 916617b07a87a0ca3525852cab0abe5b with result from task 68 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 69 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 69 try 0 failed parsl.dataflow.dflow INFO: Task 69 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/bdba3bcc-62b9-4863-8016-e42a9ad562d3_calibrate_855910_49.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/bdba3bcc-62b9-4863-8016-e42a9ad562d3_calibrate_855910_49.stderr parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/1ec4b3f8-4b13-4fee-b48a-47c0fa0a0780_characterizeImage_855910_49.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/1ec4b3f8-4b13-4fee-b48a-47c0fa0a0780_characterizeImage_855910_49.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 110 try 1 failed parsl.dataflow.dflow ERROR: Task 110 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 40ae38374454a53ca561820a6f98841b with result from task 110 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 111 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 111 try 0 failed parsl.dataflow.dflow INFO: Task 111 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/ded9a00d-cf18-403c-a850-a1cf1512a4a7_calibrate_855910_2.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/ded9a00d-cf18-403c-a850-a1cf1512a4a7_calibrate_855910_2.stderr parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/998f5281-04bd-4d22-8886-5c8fb108044a_characterizeImage_855910_2.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/998f5281-04bd-4d22-8886-5c8fb108044a_characterizeImage_855910_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 76 try 1 failed parsl.dataflow.dflow ERROR: Task 76 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 1f8a55e3adbb84f41a38bdb1ec792207 with result from task 76 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 77 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 77 try 0 failed parsl.dataflow.dflow INFO: Task 77 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/197f1e73-23b7-42e0-bbf8-e67a4c123940_calibrate_855911_44.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/197f1e73-23b7-42e0-bbf8-e67a4c123940_calibrate_855911_44.stderr parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/c39b5901-952f-4dbb-8429-4784cd1d4b0b_characterizeImage_855911_44.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/c39b5901-952f-4dbb-8429-4784cd1d4b0b_characterizeImage_855911_44.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 86 try 1 failed parsl.dataflow.dflow ERROR: Task 86 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry ecc45e6170d2818c2679a3b16a875ceb with result from task 86 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 87 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 87 try 0 failed parsl.dataflow.dflow INFO: Task 87 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/80e55c9f-ddf0-463f-b9c0-c318e595f7ee_calibrate_855911_40.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/80e55c9f-ddf0-463f-b9c0-c318e595f7ee_calibrate_855911_40.stderr parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/ba5b3c5b-c69c-4f23-a638-8d109708f1f9_characterizeImage_855911_40.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/ba5b3c5b-c69c-4f23-a638-8d109708f1f9_characterizeImage_855911_40.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 82 try 1 failed parsl.dataflow.dflow ERROR: Task 82 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 56412489eeec511593cf20792cabf282 with result from task 82 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 83 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 83 try 0 failed parsl.dataflow.dflow INFO: Task 83 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/71700be1-3e61-4a40-b3f9-c8a7b9ed9b56_calibrate_855910_28.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/71700be1-3e61-4a40-b3f9-c8a7b9ed9b56_calibrate_855910_28.stderr parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/905952c7-5ff6-4620-bf10-5a34eb50bbfe_characterizeImage_855910_28.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/905952c7-5ff6-4620-bf10-5a34eb50bbfe_characterizeImage_855910_28.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 88 try 1 failed parsl.dataflow.dflow ERROR: Task 88 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 9a8a62e864cf0fd8f5f3e85efdb9fd59 with result from task 88 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 89 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 89 try 0 failed parsl.dataflow.dflow INFO: Task 89 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/9dcb2ea2-5a8d-4fce-8fac-e6747e44ee1d_calibrate_855910_5.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/9dcb2ea2-5a8d-4fce-8fac-e6747e44ee1d_calibrate_855910_5.stderr parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/482b7420-a0fa-49f9-9de2-7fb3d7a1456b_characterizeImage_855910_5.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/482b7420-a0fa-49f9-9de2-7fb3d7a1456b_characterizeImage_855910_5.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 92 try 1 failed parsl.dataflow.dflow ERROR: Task 92 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry e668e5029f3cc4355d61e39ce062ae33 with result from task 92 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 93 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 93 try 0 failed parsl.dataflow.dflow INFO: Task 93 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/ee35fee5-76db-4249-b067-d1999129a8a1_calibrate_855910_40.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/ee35fee5-76db-4249-b067-d1999129a8a1_calibrate_855910_40.stderr parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/336be782-3ef6-4c4e-8fff-25113c3002d8_characterizeImage_855910_40.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/336be782-3ef6-4c4e-8fff-25113c3002d8_characterizeImage_855910_40.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 90 try 1 failed parsl.dataflow.dflow ERROR: Task 90 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry c9f7122a99cb5f462baba291a37979ca with result from task 90 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 91 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 91 try 0 failed parsl.dataflow.dflow INFO: Task 91 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/cf79fc9e-6f0b-4d9c-b957-9a0d3815313b_calibrate_855910_45.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/cf79fc9e-6f0b-4d9c-b957-9a0d3815313b_calibrate_855910_45.stderr parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/c88e7d7f-9e9f-494d-9118-e4e91bb318b3_characterizeImage_855910_45.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/c88e7d7f-9e9f-494d-9118-e4e91bb318b3_characterizeImage_855910_45.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 132 try 1 failed parsl.dataflow.dflow ERROR: Task 132 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry b2f76cb283beaf8b0d7e287f6c52b1f4 with result from task 132 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 133 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 133 try 0 failed parsl.dataflow.dflow INFO: Task 133 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855821/8a9abeb1-0aa0-45fc-a58e-e6b098f423ff_calibrate_855821_2.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855821/8a9abeb1-0aa0-45fc-a58e-e6b098f423ff_calibrate_855821_2.stderr parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855821/4c2d509c-480b-4b38-8207-7abd607c2208_characterizeImage_855821_2.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855821/4c2d509c-480b-4b38-8207-7abd607c2208_characterizeImage_855821_2.stderr 2024-03-13 01:51:21 proc_lsst.multi:146 [INFO] found job 23564 in provider local proc_lsst.multi INFO: found job 23564 in provider local 2024-03-13 01:51:21 proc_lsst.multi:146 [INFO] found job 17006978 in provider astro proc_lsst.multi INFO: found job 17006978 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 84 try 1 failed parsl.dataflow.dflow ERROR: Task 84 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 7db659a4949028ff9d17569a29b879b0 with result from task 84 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 85 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 85 try 0 failed parsl.dataflow.dflow INFO: Task 85 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/4beb5565-5277-42ea-9d3c-3aecfb487588_calibrate_855911_29.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/4beb5565-5277-42ea-9d3c-3aecfb487588_calibrate_855911_29.stderr parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/5a8f3d07-cc65-42a3-8325-39d418efaa04_characterizeImage_855911_29.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/5a8f3d07-cc65-42a3-8325-39d418efaa04_characterizeImage_855911_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:51:21 proc_lsst.multi:146 [INFO] found job 17006979 in provider astro proc_lsst.multi INFO: found job 17006979 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 102 try 1 failed parsl.dataflow.dflow ERROR: Task 102 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry ab3806fbb694140aa958f8e3ef225979 with result from task 102 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 103 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 103 try 0 failed parsl.dataflow.dflow INFO: Task 103 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/92912ab5-5bcc-4751-9c66-de1a91410ba2_calibrate_855910_19.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/92912ab5-5bcc-4751-9c66-de1a91410ba2_calibrate_855910_19.stderr parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/bd8244e3-e062-4ec0-a4ed-f59d0d413940_characterizeImage_855910_19.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/bd8244e3-e062-4ec0-a4ed-f59d0d413940_characterizeImage_855910_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:51:22 proc_lsst.multi:146 [INFO] found job 17006980 in provider astro proc_lsst.multi INFO: found job 17006980 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 98 try 1 failed parsl.dataflow.dflow ERROR: Task 98 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry debbfeca5e01ca676be2aa833fe92b16 with result from task 98 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 99 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 99 try 0 failed parsl.dataflow.dflow INFO: Task 99 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/a5f1faca-d945-480f-90e7-6804b76d25f9_calibrate_855910_15.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/a5f1faca-d945-480f-90e7-6804b76d25f9_calibrate_855910_15.stderr parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/fcf9a600-65ab-454e-9347-efced3f152be_characterizeImage_855910_15.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/fcf9a600-65ab-454e-9347-efced3f152be_characterizeImage_855910_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:51:23 proc_lsst.multi:146 [INFO] found job 17006981 in provider astro proc_lsst.multi INFO: found job 17006981 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 106 try 1 failed parsl.dataflow.dflow ERROR: Task 106 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 3214a44fb0bc6e3a79673fbcc3396199 with result from task 106 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 107 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 107 try 0 failed parsl.dataflow.dflow INFO: Task 107 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/55217cb4-62fe-4771-824e-32f015dab5c5_calibrate_855911_31.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/55217cb4-62fe-4771-824e-32f015dab5c5_calibrate_855911_31.stderr parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/52cac3f5-dbc4-475d-b6b6-bae04ea6ee76_characterizeImage_855911_31.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/52cac3f5-dbc4-475d-b6b6-bae04ea6ee76_characterizeImage_855911_31.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 94 try 1 failed parsl.dataflow.dflow ERROR: Task 94 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry d95271db41ba59854c87f2cbba13d89f with result from task 94 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 95 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 95 try 0 failed parsl.dataflow.dflow INFO: Task 95 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/20fe226f-7f74-4d09-9cde-bc235583eb11_calibrate_855911_23.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/20fe226f-7f74-4d09-9cde-bc235583eb11_calibrate_855911_23.stderr parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/50ff64ee-d8fa-4799-b889-922042a7be83_characterizeImage_855911_23.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/50ff64ee-d8fa-4799-b889-922042a7be83_characterizeImage_855911_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:51:23 proc_lsst.multi:146 [INFO] found job 17006982 in provider astro proc_lsst.multi INFO: found job 17006982 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:51:24 proc_lsst.multi:146 [INFO] found job 17006983 in provider astro proc_lsst.multi INFO: found job 17006983 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:51:24 proc_lsst.multi:146 [INFO] found job 17006984 in provider astro proc_lsst.multi INFO: found job 17006984 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 100 try 1 failed parsl.dataflow.dflow ERROR: Task 100 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 12f48c0cf5a2887697ac9554a734c4fe with result from task 100 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 101 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 101 try 0 failed parsl.dataflow.dflow INFO: Task 101 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/bff4aa5c-a47d-43b1-a945-b062cd66fa00_calibrate_855910_8.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/bff4aa5c-a47d-43b1-a945-b062cd66fa00_calibrate_855910_8.stderr parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/6766e172-c6fd-4873-9723-cd77771eb1bd_characterizeImage_855910_8.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/6766e172-c6fd-4873-9723-cd77771eb1bd_characterizeImage_855910_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:51:25 proc_lsst.multi:146 [INFO] found job 17006985 in provider astro proc_lsst.multi INFO: found job 17006985 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:51:25 proc_lsst.multi:146 [INFO] found job 17006986 in provider astro proc_lsst.multi INFO: found job 17006986 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 108 try 1 failed parsl.dataflow.dflow ERROR: Task 108 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 9d8f97b8e1cde209a2efeffc8ae6480a with result from task 108 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 109 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 109 try 0 failed parsl.dataflow.dflow INFO: Task 109 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/424f5474-0bf9-40b2-87c7-abc64e897f98_calibrate_855911_17.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/424f5474-0bf9-40b2-87c7-abc64e897f98_calibrate_855911_17.stderr parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/ab699881-3e42-4cd7-aaea-c25fb4618495_characterizeImage_855911_17.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/ab699881-3e42-4cd7-aaea-c25fb4618495_characterizeImage_855911_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:51:26 proc_lsst.multi:146 [INFO] found job 17006987 in provider astro proc_lsst.multi INFO: found job 17006987 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:51:26 proc_lsst.multi:146 [INFO] found job 17006988 in provider astro proc_lsst.multi INFO: found job 17006988 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 12 try 1 failed parsl.dataflow.dflow ERROR: Task 12 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry b7bcf3fad9715cc4965aac2677681012 with result from task 12 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/d136b04a-8eac-4050-8d12-d225ce3c1cf0_calibrate_855910_35.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/d136b04a-8eac-4050-8d12-d225ce3c1cf0_calibrate_855910_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:51:27 proc_lsst.multi:146 [INFO] found job 17006989 in provider astro proc_lsst.multi INFO: found job 17006989 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 120 try 1 failed parsl.dataflow.dflow ERROR: Task 120 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 7e22f0114c0ed233b4b693665ac93125 with result from task 120 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 121 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 121 try 0 failed parsl.dataflow.dflow INFO: Task 121 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/f1aa185a-52e8-4276-934f-b40197f892d2_calibrate_855910_20.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/f1aa185a-52e8-4276-934f-b40197f892d2_calibrate_855910_20.stderr parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/557817d1-e506-4563-a201-83fb89cacc81_characterizeImage_855910_20.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/557817d1-e506-4563-a201-83fb89cacc81_characterizeImage_855910_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:51:27 proc_lsst.multi:146 [INFO] found job 17006990 in provider astro proc_lsst.multi INFO: found job 17006990 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 124 try 1 failed parsl.dataflow.dflow ERROR: Task 124 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 783324ae5e67dbaeabe63595c1dd6d5e with result from task 124 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 125 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 125 try 0 failed parsl.dataflow.dflow INFO: Task 125 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/060a5edf-e4a4-40ba-aad5-ebd421c1aeea_calibrate_855911_20.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/060a5edf-e4a4-40ba-aad5-ebd421c1aeea_calibrate_855911_20.stderr parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/76337c37-3ebd-49be-b1db-586b608487b7_characterizeImage_855911_20.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/76337c37-3ebd-49be-b1db-586b608487b7_characterizeImage_855911_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:51:27 proc_lsst.multi:146 [INFO] found job 17006991 in provider astro proc_lsst.multi INFO: found job 17006991 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 114 try 1 failed parsl.dataflow.dflow ERROR: Task 114 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry c27671771e741ba6ba212498da98a89a with result from task 114 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 115 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 115 try 0 failed parsl.dataflow.dflow INFO: Task 115 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/816133dc-5e5b-47fa-a402-62ac5bd98ffc_calibrate_855911_4.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/816133dc-5e5b-47fa-a402-62ac5bd98ffc_calibrate_855911_4.stderr parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/5ac1ce51-aedf-4f82-9db8-8ec41a8abf75_characterizeImage_855911_4.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/5ac1ce51-aedf-4f82-9db8-8ec41a8abf75_characterizeImage_855911_4.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 116 try 1 failed parsl.dataflow.dflow ERROR: Task 116 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry b8841e91316276e5dd6891d38acb698c with result from task 116 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 117 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 117 try 0 failed parsl.dataflow.dflow INFO: Task 117 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/d127e2e8-3c45-4059-b2d8-d6ee832c5647_calibrate_855911_46.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/d127e2e8-3c45-4059-b2d8-d6ee832c5647_calibrate_855911_46.stderr parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/ba6bd293-3d10-4464-ba8d-d203f2b41a51_characterizeImage_855911_46.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/ba6bd293-3d10-4464-ba8d-d203f2b41a51_characterizeImage_855911_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:51:28 proc_lsst.multi:146 [INFO] found job 17006992 in provider astro proc_lsst.multi INFO: found job 17006992 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:51:28 proc_lsst.multi:146 [INFO] found job 17006993 in provider astro proc_lsst.multi INFO: found job 17006993 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 7 try 1 failed parsl.dataflow.dflow ERROR: Task 7 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry a4de438e871dead9fe871ac68864ef3c with result from task 7 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/235a9997-cb8d-4b9d-927a-026f2e0b49dd_calibrate_855911_41.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/235a9997-cb8d-4b9d-927a-026f2e0b49dd_calibrate_855911_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:51:28 proc_lsst.multi:146 [INFO] found job 17006994 in provider astro proc_lsst.multi INFO: found job 17006994 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:51:28 proc_lsst.multi:146 [INFO] found job 17006995 in provider astro proc_lsst.multi INFO: found job 17006995 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING 2024-03-13 01:51:29 proc_lsst.multi:146 [INFO] found job 17006996 in provider astro proc_lsst.multi INFO: found job 17006996 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17006978,17006979,17006980,17006981,17006982,17006983,17006984,17006985,17006986,17006987,17006988,17006989,17006990,17006991,17006992,17006993,17006994,17006995,17006996' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 130 try 1 failed parsl.dataflow.dflow ERROR: Task 130 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 4f1947c73e07e6a03abda598178896eb with result from task 130 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 131 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 131 try 0 failed parsl.dataflow.dflow INFO: Task 131 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/2ac218e4-d0f7-41d9-9228-3ef453503c46_calibrate_855911_33.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/2ac218e4-d0f7-41d9-9228-3ef453503c46_calibrate_855911_33.stderr parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/530393c1-16cf-4683-9451-8f61161fb9ab_characterizeImage_855911_33.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/530393c1-16cf-4683-9451-8f61161fb9ab_characterizeImage_855911_33.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 118 try 1 failed parsl.dataflow.dflow ERROR: Task 118 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 6d57f783396e1d260a5a5ee9cd4bdb2c with result from task 118 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 119 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 119 try 0 failed parsl.dataflow.dflow INFO: Task 119 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/c54741e7-1d23-4204-be1b-40baffe978d5_calibrate_855910_33.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/c54741e7-1d23-4204-be1b-40baffe978d5_calibrate_855910_33.stderr parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/3f32effb-4971-4cd6-a6fb-07c39cefa4e0_characterizeImage_855910_33.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/3f32effb-4971-4cd6-a6fb-07c39cefa4e0_characterizeImage_855910_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17006995 R 17006996 R 17006990 R 17006991 R 17006992 R 17006993 R 17006994 R 17006988 R 17006989 R 17006983 R 17006984 R 17006985 R 17006986 R 17006987 R 17006982 R 17006978 R 17006979 R 17006980 R 17006981 R parsl.providers.slurm.slurm DEBUG: Updating job 17006995 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006996 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006990 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006991 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006992 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006993 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006994 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006988 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006989 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006983 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006984 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006985 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006986 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006987 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006982 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006978 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006979 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006980 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17006981 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 20, active_tasks = 9 parsl.dataflow.strategy DEBUG: Executor multi has 9 active tasks, 20/0 running/pending blocks, and 21 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22667511342224 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 20, active_tasks = 9 parsl.dataflow.strategy DEBUG: Executor multi has 9 active tasks, 20/0 running/pending blocks, and 21 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22667511342224 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 122 try 1 failed parsl.dataflow.dflow ERROR: Task 122 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry f08c8251214a6eb92bf951472fb4501e with result from task 122 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 123 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 123 try 0 failed parsl.dataflow.dflow INFO: Task 123 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/c9db1897-ae21-4f2d-9f79-c24cade6454e_calibrate_855911_47.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/c9db1897-ae21-4f2d-9f79-c24cade6454e_calibrate_855911_47.stderr parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/b24378e6-0128-4e39-8954-9690a15457bd_characterizeImage_855911_47.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/b24378e6-0128-4e39-8954-9690a15457bd_characterizeImage_855911_47.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 138 try 1 failed parsl.dataflow.dflow ERROR: Task 138 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 8c00d28bc925c30b332208303fa03956 with result from task 138 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 139 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 139 try 0 failed parsl.dataflow.dflow INFO: Task 139 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/2cb49ae3-0a7f-459c-a3ce-87203f1fbddb_calibrate_855911_8.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/2cb49ae3-0a7f-459c-a3ce-87203f1fbddb_calibrate_855911_8.stderr parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/7ed6d274-7055-4b37-9f85-83443fae8a1f_characterizeImage_855911_8.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/7ed6d274-7055-4b37-9f85-83443fae8a1f_characterizeImage_855911_8.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 136 try 1 failed parsl.dataflow.dflow ERROR: Task 136 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 85c07d4797533be7aacba981040dfd08 with result from task 136 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 137 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 137 try 0 failed parsl.dataflow.dflow INFO: Task 137 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/8cc6a515-d09c-4249-af24-abfba9e8dbcd_calibrate_855911_19.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/8cc6a515-d09c-4249-af24-abfba9e8dbcd_calibrate_855911_19.stderr parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/0c0696ad-bcb9-4ee0-bd31-858ea5955eee_characterizeImage_855911_19.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/0c0696ad-bcb9-4ee0-bd31-858ea5955eee_characterizeImage_855911_19.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 126 try 1 failed parsl.dataflow.dflow ERROR: Task 126 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 6b781062ea895e02e9694ec3fd22d6dd with result from task 126 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 127 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 127 try 0 failed parsl.dataflow.dflow INFO: Task 127 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/ff7c57fb-10f5-4b43-8567-e463647cd25b_calibrate_855910_39.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/ff7c57fb-10f5-4b43-8567-e463647cd25b_calibrate_855910_39.stderr parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/75228b8e-af24-4ef7-a2ab-6ad77397b1f6_characterizeImage_855910_39.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/75228b8e-af24-4ef7-a2ab-6ad77397b1f6_characterizeImage_855910_39.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 104 try 1 failed parsl.dataflow.dflow ERROR: Task 104 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 04c191a1fe343b963abedb882721781f with result from task 104 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 105 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 105 try 0 failed parsl.dataflow.dflow INFO: Task 105 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/c7e54559-3539-48fc-ba48-7cd24f608b6e_calibrate_855910_25.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/c7e54559-3539-48fc-ba48-7cd24f608b6e_calibrate_855910_25.stderr parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/98e11768-2250-42f8-90b4-f807d804f98c_characterizeImage_855910_25.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/98e11768-2250-42f8-90b4-f807d804f98c_characterizeImage_855910_25.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 128 try 1 failed parsl.dataflow.dflow ERROR: Task 128 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 547525b8c5361f46105c6721959871a8 with result from task 128 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 129 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 129 try 0 failed parsl.dataflow.dflow INFO: Task 129 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/b46511d0-5d10-47ab-9887-1a5b3235b6b8_calibrate_855910_13.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/b46511d0-5d10-47ab-9887-1a5b3235b6b8_calibrate_855910_13.stderr parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/b9507bfd-4907-420b-89e8-49e3d290e347_characterizeImage_855910_13.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/b9507bfd-4907-420b-89e8-49e3d290e347_characterizeImage_855910_13.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 134 try 1 failed parsl.dataflow.dflow ERROR: Task 134 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry f4b087fca69cbd10e0ea212117e8ab18 with result from task 134 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 135 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 135 try 0 failed parsl.dataflow.dflow INFO: Task 135 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/29c4d2cb-5142-4b24-bb6d-aedf07c814f8_calibrate_855911_53.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/29c4d2cb-5142-4b24-bb6d-aedf07c814f8_calibrate_855911_53.stderr parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/1ac43a9a-71de-4903-bf82-a267193afbc2_characterizeImage_855911_53.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/1ac43a9a-71de-4903-bf82-a267193afbc2_characterizeImage_855911_53.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 20, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 20/0 running/pending blocks, and 21 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22667511342224 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 112 try 1 failed parsl.dataflow.dflow ERROR: Task 112 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry e2666f3d8b9748f98ae027abbca45960 with result from task 112 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 113 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 113 try 0 failed parsl.dataflow.dflow INFO: Task 113 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/428c292f-0021-4e7b-b01c-e07fe4ab8f31_calibrate_855911_61.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855911/428c292f-0021-4e7b-b01c-e07fe4ab8f31_calibrate_855911_61.stderr parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/98714589-92e8-4855-973f-8251c4b5f8c9_characterizeImage_855911_61.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855911/98714589-92e8-4855-973f-8251c4b5f8c9_characterizeImage_855911_61.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 20, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 20/0 running/pending blocks, and 21 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22667511342224 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 20, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 20/0 running/pending blocks, and 21 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22667511342224 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 78 try 1 failed parsl.dataflow.dflow ERROR: Task 78 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 20dbd507b90cddb48c5bec60f75cd595 with result from task 78 parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Task 79 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 79 try 0 failed parsl.dataflow.dflow INFO: Task 79 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow INFO: DFK cleanup initiated parsl.dataflow.dflow INFO: Summary of tasks in DFK: parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0 parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/47618002-cc64-485f-8b8e-3e1e481a4f25_calibrate_855910_61.stdout parsl.dataflow.dflow INFO: Tasks in state States.pending: 0 parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/calibrate/855910/47618002-cc64-485f-8b8e-3e1e481a4f25_calibrate_855910_61.stderr parsl.dataflow.dflow INFO: Tasks in state States.running: 0 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/1d3d4c4e-0c48-4edb-8599-e62a2b871238_characterizeImage_855910_61.stdout parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 0 parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/submit/DEEP/20190507/A1c/science#step1/20240313T014901Z/logs/characterizeImage/855910/1d3d4c4e-0c48-4edb-8599-e62a2b871238_characterizeImage_855910_61.stderr parsl.dataflow.dflow INFO: Tasks in state States.failed: 77 parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 63 parsl.dataflow.dflow INFO: Tasks in state States.launched: 0 parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0 parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0 parsl.dataflow.dflow INFO: Tasks in state States.joining: 0 parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0 parsl.dataflow.dflow INFO: End of summary parsl.dataflow.dflow WARNING: No tasks checkpointed so far in this run. Please ensure caching is enabled parsl.dataflow.dflow INFO: Closing job status poller parsl.dataflow.dflow INFO: Terminated job status poller parsl.dataflow.dflow INFO: Scaling in and shutting down executors parsl.dataflow.dflow INFO: Scaling in executor multi parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=20, block_ids=[] parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 20 blocks parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8917a7dba34e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8917a7dba34e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0b883b2f710b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0b883b2f710b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a05123639f28 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a05123639f28 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6a7a8d59de9b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6a7a8d59de9b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 48430b9d9f1c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 48430b9d9f1c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 689982aa5289 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 689982aa5289 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b78e2c7030f0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b78e2c7030f0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 656a771637f7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 656a771637f7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ba02ee4330bf parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ba02ee4330bf parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c198f64fcc3e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c198f64fcc3e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0fb5b387f61c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0fb5b387f61c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e739ed817607 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e739ed817607 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 94ee4928b238 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 94ee4928b238 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3f09800c4c09 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3f09800c4c09 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7c327b990efd parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7c327b990efd parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 15860c4b5848 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 15860c4b5848 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 16614011f8d8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 16614011f8d8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bfffdefd6578 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bfffdefd6578 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 85ff0b61e715 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 85ff0b61e715 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1c04f9fabd52 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1c04f9fabd52 2024-03-13 01:51:49 proc_lsst.multi:146 [INFO] found job 23564 in provider local proc_lsst.multi INFO: found job 23564 in provider local 2024-03-13 01:51:49 proc_lsst.multi:201 [INFO] cancelling 23564 on provider local proc_lsst.multi INFO: cancelling 23564 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 23564 2024-03-13 01:51:49 proc_lsst.multi:146 [INFO] found job 17006986 in provider astro proc_lsst.multi INFO: found job 17006986 in provider astro 2024-03-13 01:51:49 proc_lsst.multi:201 [INFO] cancelling 17006986 on provider astro proc_lsst.multi INFO: cancelling 17006986 on provider astro 2024-03-13 01:51:49 proc_lsst.multi:146 [INFO] found job 17006988 in provider astro proc_lsst.multi INFO: found job 17006988 in provider astro 2024-03-13 01:51:49 proc_lsst.multi:201 [INFO] cancelling 17006988 on provider astro proc_lsst.multi INFO: cancelling 17006988 on provider astro 2024-03-13 01:51:49 proc_lsst.multi:146 [INFO] found job 17006979 in provider astro proc_lsst.multi INFO: found job 17006979 in provider astro 2024-03-13 01:51:49 proc_lsst.multi:201 [INFO] cancelling 17006979 on provider astro proc_lsst.multi INFO: cancelling 17006979 on provider astro 2024-03-13 01:51:49 proc_lsst.multi:146 [INFO] found job 17006978 in provider astro proc_lsst.multi INFO: found job 17006978 in provider astro 2024-03-13 01:51:49 proc_lsst.multi:201 [INFO] cancelling 17006978 on provider astro proc_lsst.multi INFO: cancelling 17006978 on provider astro 2024-03-13 01:51:49 proc_lsst.multi:146 [INFO] found job 17006982 in provider astro proc_lsst.multi INFO: found job 17006982 in provider astro 2024-03-13 01:51:49 proc_lsst.multi:201 [INFO] cancelling 17006982 on provider astro proc_lsst.multi INFO: cancelling 17006982 on provider astro 2024-03-13 01:51:49 proc_lsst.multi:146 [INFO] found job 17006987 in provider astro proc_lsst.multi INFO: found job 17006987 in provider astro 2024-03-13 01:51:49 proc_lsst.multi:201 [INFO] cancelling 17006987 on provider astro proc_lsst.multi INFO: cancelling 17006987 on provider astro 2024-03-13 01:51:49 proc_lsst.multi:146 [INFO] found job 17006996 in provider astro proc_lsst.multi INFO: found job 17006996 in provider astro 2024-03-13 01:51:49 proc_lsst.multi:201 [INFO] cancelling 17006996 on provider astro proc_lsst.multi INFO: cancelling 17006996 on provider astro 2024-03-13 01:51:50 proc_lsst.multi:146 [INFO] found job 17006984 in provider astro proc_lsst.multi INFO: found job 17006984 in provider astro 2024-03-13 01:51:50 proc_lsst.multi:201 [INFO] cancelling 17006984 on provider astro proc_lsst.multi INFO: cancelling 17006984 on provider astro 2024-03-13 01:51:50 proc_lsst.multi:146 [INFO] found job 17006980 in provider astro proc_lsst.multi INFO: found job 17006980 in provider astro 2024-03-13 01:51:50 proc_lsst.multi:201 [INFO] cancelling 17006980 on provider astro proc_lsst.multi INFO: cancelling 17006980 on provider astro 2024-03-13 01:51:50 proc_lsst.multi:146 [INFO] found job 17006994 in provider astro proc_lsst.multi INFO: found job 17006994 in provider astro 2024-03-13 01:51:50 proc_lsst.multi:201 [INFO] cancelling 17006994 on provider astro proc_lsst.multi INFO: cancelling 17006994 on provider astro 2024-03-13 01:51:50 proc_lsst.multi:146 [INFO] found job 17006983 in provider astro proc_lsst.multi INFO: found job 17006983 in provider astro 2024-03-13 01:51:50 proc_lsst.multi:201 [INFO] cancelling 17006983 on provider astro proc_lsst.multi INFO: cancelling 17006983 on provider astro 2024-03-13 01:51:50 proc_lsst.multi:146 [INFO] found job 17006993 in provider astro proc_lsst.multi INFO: found job 17006993 in provider astro 2024-03-13 01:51:50 proc_lsst.multi:201 [INFO] cancelling 17006993 on provider astro proc_lsst.multi INFO: cancelling 17006993 on provider astro 2024-03-13 01:51:50 proc_lsst.multi:146 [INFO] found job 17006995 in provider astro proc_lsst.multi INFO: found job 17006995 in provider astro 2024-03-13 01:51:50 proc_lsst.multi:201 [INFO] cancelling 17006995 on provider astro proc_lsst.multi INFO: cancelling 17006995 on provider astro 2024-03-13 01:51:51 proc_lsst.multi:146 [INFO] found job 17006985 in provider astro proc_lsst.multi INFO: found job 17006985 in provider astro 2024-03-13 01:51:51 proc_lsst.multi:201 [INFO] cancelling 17006985 on provider astro proc_lsst.multi INFO: cancelling 17006985 on provider astro 2024-03-13 01:51:51 proc_lsst.multi:146 [INFO] found job 17006991 in provider astro proc_lsst.multi INFO: found job 17006991 in provider astro 2024-03-13 01:51:51 proc_lsst.multi:201 [INFO] cancelling 17006991 on provider astro proc_lsst.multi INFO: cancelling 17006991 on provider astro 2024-03-13 01:51:51 proc_lsst.multi:146 [INFO] found job 17006992 in provider astro proc_lsst.multi INFO: found job 17006992 in provider astro 2024-03-13 01:51:51 proc_lsst.multi:201 [INFO] cancelling 17006992 on provider astro proc_lsst.multi INFO: cancelling 17006992 on provider astro 2024-03-13 01:51:51 proc_lsst.multi:146 [INFO] found job 17006981 in provider astro proc_lsst.multi INFO: found job 17006981 in provider astro 2024-03-13 01:51:51 proc_lsst.multi:201 [INFO] cancelling 17006981 on provider astro proc_lsst.multi INFO: cancelling 17006981 on provider astro 2024-03-13 01:51:51 proc_lsst.multi:146 [INFO] found job 17006990 in provider astro proc_lsst.multi INFO: found job 17006990 in provider astro 2024-03-13 01:51:51 proc_lsst.multi:201 [INFO] cancelling 17006990 on provider astro proc_lsst.multi INFO: cancelling 17006990 on provider astro 2024-03-13 01:51:51 proc_lsst.multi:146 [INFO] found job 17006989 in provider astro proc_lsst.multi INFO: found job 17006989 in provider astro 2024-03-13 01:51:51 proc_lsst.multi:201 [INFO] cancelling 17006989 on provider astro proc_lsst.multi INFO: cancelling 17006989 on provider astro parsl.dataflow.dflow INFO: Shutting down executor multi 2024-03-13 01:51:52 proc_lsst.multi:40 [INFO] Cancelling all provider resources proc_lsst.multi INFO: Cancelling all provider resources 2024-03-13 01:51:52 proc_lsst.multi:47 [INFO] new jobs since last cancel ['17006978', '17006979', '17006980', '17006981', '17006982', '17006983', '17006984', '17006985', '17006986', '17006987', '17006988', '17006989', '17006990', '17006991', '17006992', '17006993', '17006994', '17006995', '17006996', '23564'] proc_lsst.multi INFO: new jobs since last cancel ['17006978', '17006979', '17006980', '17006981', '17006982', '17006983', '17006984', '17006985', '17006986', '17006987', '17006988', '17006989', '17006990', '17006991', '17006992', '17006993', '17006994', '17006995', '17006996', '23564'] 2024-03-13 01:51:52 proc_lsst.multi:146 [INFO] found job 17006978 in provider astro proc_lsst.multi INFO: found job 17006978 in provider astro 2024-03-13 01:51:52 proc_lsst.multi:201 [INFO] cancelling 17006978 on provider astro proc_lsst.multi INFO: cancelling 17006978 on provider astro 2024-03-13 01:51:52 proc_lsst.multi:146 [INFO] found job 17006979 in provider astro proc_lsst.multi INFO: found job 17006979 in provider astro 2024-03-13 01:51:52 proc_lsst.multi:201 [INFO] cancelling 17006979 on provider astro proc_lsst.multi INFO: cancelling 17006979 on provider astro 2024-03-13 01:51:52 proc_lsst.multi:146 [INFO] found job 17006980 in provider astro proc_lsst.multi INFO: found job 17006980 in provider astro 2024-03-13 01:51:52 proc_lsst.multi:201 [INFO] cancelling 17006980 on provider astro proc_lsst.multi INFO: cancelling 17006980 on provider astro 2024-03-13 01:51:52 proc_lsst.multi:146 [INFO] found job 17006981 in provider astro proc_lsst.multi INFO: found job 17006981 in provider astro 2024-03-13 01:51:52 proc_lsst.multi:201 [INFO] cancelling 17006981 on provider astro proc_lsst.multi INFO: cancelling 17006981 on provider astro 2024-03-13 01:51:52 proc_lsst.multi:146 [INFO] found job 17006982 in provider astro proc_lsst.multi INFO: found job 17006982 in provider astro 2024-03-13 01:51:52 proc_lsst.multi:201 [INFO] cancelling 17006982 on provider astro proc_lsst.multi INFO: cancelling 17006982 on provider astro 2024-03-13 01:51:52 proc_lsst.multi:146 [INFO] found job 17006983 in provider astro proc_lsst.multi INFO: found job 17006983 in provider astro 2024-03-13 01:51:52 proc_lsst.multi:201 [INFO] cancelling 17006983 on provider astro proc_lsst.multi INFO: cancelling 17006983 on provider astro 2024-03-13 01:51:52 proc_lsst.multi:146 [INFO] found job 17006984 in provider astro proc_lsst.multi INFO: found job 17006984 in provider astro 2024-03-13 01:51:52 proc_lsst.multi:201 [INFO] cancelling 17006984 on provider astro proc_lsst.multi INFO: cancelling 17006984 on provider astro 2024-03-13 01:51:53 proc_lsst.multi:146 [INFO] found job 17006985 in provider astro proc_lsst.multi INFO: found job 17006985 in provider astro 2024-03-13 01:51:53 proc_lsst.multi:201 [INFO] cancelling 17006985 on provider astro proc_lsst.multi INFO: cancelling 17006985 on provider astro 2024-03-13 01:51:53 proc_lsst.multi:146 [INFO] found job 17006986 in provider astro proc_lsst.multi INFO: found job 17006986 in provider astro 2024-03-13 01:51:53 proc_lsst.multi:201 [INFO] cancelling 17006986 on provider astro proc_lsst.multi INFO: cancelling 17006986 on provider astro 2024-03-13 01:51:53 proc_lsst.multi:146 [INFO] found job 17006987 in provider astro proc_lsst.multi INFO: found job 17006987 in provider astro 2024-03-13 01:51:53 proc_lsst.multi:201 [INFO] cancelling 17006987 on provider astro proc_lsst.multi INFO: cancelling 17006987 on provider astro 2024-03-13 01:51:53 proc_lsst.multi:146 [INFO] found job 17006988 in provider astro proc_lsst.multi INFO: found job 17006988 in provider astro 2024-03-13 01:51:53 proc_lsst.multi:201 [INFO] cancelling 17006988 on provider astro proc_lsst.multi INFO: cancelling 17006988 on provider astro 2024-03-13 01:51:53 proc_lsst.multi:146 [INFO] found job 17006989 in provider astro proc_lsst.multi INFO: found job 17006989 in provider astro 2024-03-13 01:51:53 proc_lsst.multi:201 [INFO] cancelling 17006989 on provider astro proc_lsst.multi INFO: cancelling 17006989 on provider astro 2024-03-13 01:51:53 proc_lsst.multi:146 [INFO] found job 17006990 in provider astro proc_lsst.multi INFO: found job 17006990 in provider astro 2024-03-13 01:51:53 proc_lsst.multi:201 [INFO] cancelling 17006990 on provider astro proc_lsst.multi INFO: cancelling 17006990 on provider astro 2024-03-13 01:51:54 proc_lsst.multi:146 [INFO] found job 17006991 in provider astro proc_lsst.multi INFO: found job 17006991 in provider astro 2024-03-13 01:51:54 proc_lsst.multi:201 [INFO] cancelling 17006991 on provider astro proc_lsst.multi INFO: cancelling 17006991 on provider astro 2024-03-13 01:51:54 proc_lsst.multi:146 [INFO] found job 17006992 in provider astro proc_lsst.multi INFO: found job 17006992 in provider astro 2024-03-13 01:51:54 proc_lsst.multi:201 [INFO] cancelling 17006992 on provider astro proc_lsst.multi INFO: cancelling 17006992 on provider astro 2024-03-13 01:51:54 proc_lsst.multi:146 [INFO] found job 17006993 in provider astro proc_lsst.multi INFO: found job 17006993 in provider astro 2024-03-13 01:51:54 proc_lsst.multi:201 [INFO] cancelling 17006993 on provider astro proc_lsst.multi INFO: cancelling 17006993 on provider astro 2024-03-13 01:51:54 proc_lsst.multi:146 [INFO] found job 17006994 in provider astro proc_lsst.multi INFO: found job 17006994 in provider astro 2024-03-13 01:51:54 proc_lsst.multi:201 [INFO] cancelling 17006994 on provider astro proc_lsst.multi INFO: cancelling 17006994 on provider astro 2024-03-13 01:51:54 proc_lsst.multi:146 [INFO] found job 17006995 in provider astro proc_lsst.multi INFO: found job 17006995 in provider astro 2024-03-13 01:51:54 proc_lsst.multi:201 [INFO] cancelling 17006995 on provider astro proc_lsst.multi INFO: cancelling 17006995 on provider astro 2024-03-13 01:51:54 proc_lsst.multi:146 [INFO] found job 17006996 in provider astro proc_lsst.multi INFO: found job 17006996 in provider astro 2024-03-13 01:51:54 proc_lsst.multi:201 [INFO] cancelling 17006996 on provider astro proc_lsst.multi INFO: cancelling 17006996 on provider astro 2024-03-13 01:51:54 proc_lsst.multi:146 [INFO] found job 23564 in provider local proc_lsst.multi INFO: found job 23564 in provider local 2024-03-13 01:51:54 proc_lsst.multi:201 [INFO] cancelling 23564 on provider local proc_lsst.multi INFO: cancelling 23564 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 23564 parsl.providers.local.local WARNING: Failed to kill PID: 23564 and child processes on local 2024-03-13 01:51:56 proc_lsst.multi:50 [INFO] no new jobs since last cancel, resuming executor shutdown proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt parsl.dataflow.dflow INFO: Shut down executor multi parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete parsl.executors.threads DEBUG: Done with executor shutdown parsl.dataflow.dflow INFO: Shut down executor _parsl_internal parsl.dataflow.dflow INFO: Terminated executors parsl.dataflow.dflow INFO: DFK cleanup complete parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 155.0889 seconds lsst.ctrl.bps.drivers INFO: Run 'DEEP_20190507_A1c_science#step1_20240313T014901Z' submitted for execution with id 'None' lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 155.0986 seconds; current memory usage: 0.347 Gibyte, delta: 0.002 Gibyte, peak delta: 0.002 Gibyte lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 169.9082 seconds; current memory usage: 0.347 Gibyte, delta: 0.210 Gibyte, peak delta: 0.210 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.347 Gibyte (main), 0.347 Gibyte (largest child process) Run Id: None Run Name: DEEP_20190507_A1c_science#step1_20240313T014901Z parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up real 2m52.870s user 0m39.118s sys 0m11.684s